Datasets - Prequel Import

In Prequel Import, a Dataset is a materialized cache of data representing a connected Source table or custom SQL query. Datasets are maintained in the Datalake. Every Dataset is defined along two independent dimensions: its type, which controls how Prequel Import detects changes, and its method, which controls how Prequel Import reads data from the Source.

Dataset Type

Dimension or Fact. Determines how Prequel Import detects changes.

Dataset Method

Table, SQL, or Glob. Determines how Prequel Import reads from the Source.

1. Dataset Type

A Dataset’s type determines which columns Prequel Import uses to detect changes and whether it can sync incrementally. It applies the same way to every method. See Change Detection for how each type is synced.

Dimension
Fact

Dimension tables contain rows that may be updated over time. These typically represent entities such as users, products, or accounts. Prequel Import tracks changes and syncs updates to existing records.Configuration fieldsSee the API reference for the full payload.

primary_key_column

string

required

The column used to uniquely identify records.

last_modified_column

string

The column Prequel Import uses to find rows changed since the last sync. When set, every sync after the first reads only rows modified since the previous run. When unset, every source row is read on each sync. Either way records are merged by primary_key_column, so this setting controls whether the source read is filtered by time, not whether rows are replaced or removed.

is_deleted_column

string

A boolean column the Source uses to flag deleted rows. Prequel Import reads this flag and propagates those deletions to the Datalake. Rows that are physically removed from the Source are not detected, whether or not this column is set.

2. Dataset Method

A Dataset’s method determines how Prequel Import reads data from the Source.

Table
SQL
Glob

Reads a named table from the Source, addressed by namespace and table name.Configuration fieldsSee the API reference for the full payload.

source_table_namespace

string

required

The namespace (for example schema or database) containing the source table.

source_table_name

string

required

The name of the table in the Source.

Reads the result set of a custom SELECT query you provide, so you can join, filter, or reshape data before it is replicated.Configuration fieldsSee the API reference for the full payload.

query

string

required

The read-only SELECT statement Prequel Import runs against the Source.

Prequel Import reads your query as a subquery, so a few constraints apply:

The query must return a result set. Use a SELECT or another read-only statement that returns rows. Statements that do not return rows, such as INSERT or UPDATE, are not supported.
Every column named in your type configuration (primary_key_column, last_modified_column, is_deleted_column, created_at_column) must appear in the query’s result set. If a configured column is missing, the extract fails.
Rows with a null primary key are skipped, so make sure your query returns a non-null primary key for every row you want to ingest.

For an incremental Dimension Dataset, the last_modified_column value must be stable and monotonically increasing so change detection advances correctly. Computed or non-monotonic timestamps can cause rows to be missed or re-read.

Example

{
  "dataset": {
    "source_id": "1f3c9e7a-0b2d-4a8e-9c11-7d6f2b5a4c33",
    "method": "sql",
    "type": "dimension",
    "frequency_minutes": 60,
    "sql": {
      "query": "SELECT p.id, p.name, p.updated_at, c.label AS category FROM products p JOIN categories c ON c.id = p.category_id WHERE p.active = true"
    },
    "dimension": {
      "primary_key_column": "id",
      "last_modified_column": "updated_at"
    }
  }
}

Reads object files that match a pattern. Available for object storage Sources.Configuration fieldsSee the API reference for the full payload.

pattern

string

required

The glob pattern matching object files in the Source.

format

string

required

The format of the object files. One of csv, parquet, jsonl, json.

Creating a Dataset

A Dataset is created with POST /import/datasets. Along with one type block and one method block from above, every Dataset specifies these common fields. Configuration fields See the API reference for the full payload.

name

string

A name for the Dataset.

source_id

uuid

required

The ID of the Source this Dataset reads from.

method

string

required

How Prequel Import reads from the Source. One of table, sql, glob.

type

string

required

The type of Dataset. One of dimension, fact.

frequency_minutes

integer

required

How often (in minutes) Prequel Import checks this Dataset for changes. Must be at least 1.

is_enabled

boolean

Whether the Dataset is enabled. Defaults to true when omitted.

Updating a Dataset

After a Dataset is created, only its name, frequency_minutes, and is_enabled can be changed, with PATCH /import/datasets/{id}. The type, method, query, and column configuration are fixed when the Dataset is created. To change a query or column mapping, create a new Dataset.

Dataset Type

Dataset Method

​1. Dataset Type

​2. Dataset Method

​Creating a Dataset

​Updating a Dataset

1. Dataset Type

2. Dataset Method

Creating a Dataset

Updating a Dataset