Types of Datasets
For efficiency, Prequel Import requires each Dataset to be assigned a type: Dimension or Fact.- Dimension: A table containing rows that may be updated over time (e.g., users, products, accounts). Prequel Import tracks changes and syncs updates to existing records.
- Fact: An append-only table containing immutable events or transactions (e.g., events, page views, logs). New records are added but existing records are never modified. Prequel Import assumes records that have been successfully delivered to the Destination do not need to be monitored for changes.
Dataset configuration
| Field | Type | Required | Description |
|---|---|---|---|
source_id | uuid | Yes | The ID of the Source this Dataset is connected to. |
source_object_namespace | string | Yes | The namespace (e.g., schema or database) containing the source table. |
source_object_name | string | Yes | The name of the table or object in the Source. |
type | string | Yes | The type of Dataset. One of dimension, fact. |
frequency_minutes | integer | Yes | How often (in minutes) Prequel checks this Dataset for changes. |
dimension | object | Required if type is dimension | Configuration for dimension datasets. See below. |
fact | object | Required if type is fact | Configuration for fact datasets. See below. |
Dimension configuration
| Field | Type | Required | Description |
|---|---|---|---|
primary_key_column | string | Yes | The column used to uniquely identify records. |
last_modified_column | string | No | The column used to track when records were last modified. Used for incremental sync. Set to null for full sync. |
is_deleted_column | string | No | A column that flags soft-deleted records. |
Fact configuration
| Field | Type | Required | Description |
|---|---|---|---|
primary_key_column | string | Yes | The column used to uniquely identify records. |
created_at_column | string | Yes | The column recording when each record was created. Required for efficient append-only change detection. |