Batch API

A Batch-based API endpoint is the recommended implementation for most use cases. Contact the Prequel team if you have questions.

A webhook_batch destination delivers multiple records in a single HTTP request. For the API call to register a destination with Prequel, including a complete example, see Create Destination.

Authentication

Prequel signs every delivery request with the X-Prequel-Webhook-Signature header so your endpoint can validate the request originated from Prequel. Fetch the public key for verification from GET /public/signatures/webhook-public-key. See Webhook headers for the full header list and verification steps. After verifying the request signature, you can optionally read provider_id and load_id from the envelope body to confirm the request belongs to an expected provider and load.

Creating your destination spec

The destination object allows you to customize the contract between data shared by your customers and the shape Prequel provides to your receiving endpoint. Example: Suppose you are importing a users table with string fields id, email, and subscription. Below are four possible shapes for a webhook_batch destination. These are not exhaustive and features can be combined to confirm to your endpoint’s requirements:

Schema as body: Records land in the JSONL artifact in the shape declared by record_schema, with no additional shaping.
Custom body: Use a body template to customize each record. Only supported when format is json.
Routed by provider: uri and/or headers vary at delivery time using the ProviderID delivery-level identifier.
Destination per provider: A separate destination is registered for each provider, with a record_schema that includes provider-specific custom fields.

Schema as body
Custom body
Routed by provider
Destination per provider

POST destination payload

{
  "name": "users-batch-destination",
  "type": "webhook_batch",
  "record_schema": {
    "type": "object",
    "properties": {
      "id":    { "type": "string" },
      "email": { "type": "string" },
      "subscription": { "type": "string" }
    },
    "required": ["id", "email"]
  },
  "webhook_batch": {
    "request_template": {
      "method": "POST",
      "uri": "https://example.com/prequel/batch",
      "headers": { "Content-Type": "application/json" }
    },
    "format": "json"
  }
}

Name your destination

Set name (string, required) to a unique identifier for this destination within your account.

Set the destination type

Set type (string, required) to webhook_batch for batch deliveries.

Declare your record schema

Set record_schema (object, required) to a JSON Schema (draft-07) document that declares the fields each record contains. Prequel validates records against this schema before delivery.

Configure the request template

Set webhook_batch.request_template (object, required) with the HTTP method (string, defaults to POST), uri (string, required), and optional headers (object). The uri and headers templates support the .Prequel.* namespace.

Pick a delivery format

Set webhook_batch.format (string, required) to one of json, csv, or parquet. See Delivery format for what each option produces.

Tune optional throughput limits

Optionally set webhook_batch.max_size_per_batch (integer, default 24000), webhook_batch.max_records_per_minute (integer, default 3000), and webhook_batch.max_concurrency (integer, default 1) to control delivery throughput.

POST destination payload

{
  "name": "users-batch-destination",
  "type": "webhook_batch",
  "record_schema": {
    "type": "object",
    "properties": {
      "id":    { "type": "string" },
      "email": { "type": "string" },
      "subscription": { "type": "string" }
    },
    "required": ["id", "email"]
  },
  "webhook_batch": {
    "request_template": {
      "method": "POST",
      "uri": "https://example.com/prequel/batch",
      "headers": { "Content-Type": "application/json" },
      "body": "{\"email\": \"{{.Record.email}}\", \"user\": {{.Record.AsJson}}, \"is_deleted\": {{.IsDeleted}}, \"provider_id\": \"{{.Prequel.ProviderID}}\", \"load_id\": \"{{.Prequel.LoadID}}\"}"
    },
    "format": "json"
  }
}

Name your destination

Set name (string, required) to a unique identifier for this destination within your account.

Set the destination type

Set type (string, required) to webhook_batch for batch deliveries.

Declare your record schema

Set record_schema (object, required) to a JSON Schema (draft-07) document that declares the fields each record contains. Prequel validates records against this schema before delivery.

Configure the request template with a body template

Set webhook_batch.request_template (object, required) with method, uri, headers, and body. The body template renders once per record while Prequel encodes the JSONL artifact, with access to record fields, the deletion flag, and delivery-level identifiers. The uri and headers templates render once per delivery and have access to delivery-level identifiers only. See Template variables for the full list of available variables. Body templates are only valid when format is json; selecting csv or parquet rejects the destination.

Pick a delivery format

Set webhook_batch.format (string, required) to one of json, csv, or parquet. See Delivery format for what each option produces.

Tune optional throughput limits

POST destination payload

{
  "name": "users-batch-destination",
  "type": "webhook_batch",
  "record_schema": {
    "type": "object",
    "properties": {
      "id":    { "type": "string" },
      "email": { "type": "string" },
      "subscription": { "type": "string" }
    },
    "required": ["id", "email"]
  },
  "webhook_batch": {
    "request_template": {
      "method": "POST",
      "uri": "https://example.com/prequel/batch?provider={{.Prequel.ProviderID}}",
      "headers": { "Content-Type": "application/json" }
    },
    "format": "json"
  }
}

Name your destination

Set name (string, required) to a unique identifier for this destination within your account.

Set the destination type

Set type (string, required) to webhook_batch for batch deliveries.

Declare your record schema

Set record_schema (object, required) to a JSON Schema (draft-07) document that declares the fields each record contains. Prequel validates records against this schema before delivery.

Route by provider in the request template

Set webhook_batch.request_template (object, required) with method, uri, and optional headers. Use {{.Prequel.ProviderID}} in uri and headers to route by the provider whose data is being delivered. These templates render once per delivery, before any records are selected. See Template variables for the full list of identifiers available.

Pick a delivery format

Set webhook_batch.format (string, required) to one of json, csv, or parquet. See Delivery format for what each option produces.

Tune optional throughput limits

POST destination payload

{
  "name": "acme-corp-batch-destination",
  "type": "webhook_batch",
  "record_schema": {
    "type": "object",
    "properties": {
      "id":      { "type": "string" },
      "email":   { "type": "string" },
      "subscription": { "type": "string" },
      "tax_id":  { "type": "string" }
    },
    "required": ["id", "email", "tax_id"]
  },
  "webhook_batch": {
    "request_template": {
      "method": "POST",
      "uri": "https://example.com/prequel/batch/acme-corp",
      "headers": { "Content-Type": "application/json" }
    },
    "format": "json"
  }
}

Create a separate destination for each provider whose requirements differ. Give each one a name (string, required) that identifies the provider it serves.

Set the destination type

Set type (string, required) to webhook_batch for batch deliveries.

Declare a provider-specific record schema

Set record_schema (object, required) to a JSON Schema (draft-07) document that declares the fields this provider sends, including any custom fields unique to them. Prequel validates records against this schema before delivery.

Configure the request template

Set webhook_batch.request_template (object, required) with method, uri, and optional headers. Because the destination is scoped to a single provider, you can hard-code provider-specific routing into uri and headers without needing template variables.

Pick a delivery format

Set webhook_batch.format (string, required) to one of json, csv, or parquet. See Delivery format for what each option produces.

Tune optional throughput limits

Template variables

The body template has access to every variable below; the uri and headers templates only have access to the .Prequel.* namespace.

Variable	Type	Description
`{{.Record.<field>}}`	any	Value of a specific field from the record (e.g. `{{.Record.email}}`). The field must be declared in `record_schema`.
`{{.Record.AsJson}}`	object	The full record serialized as a JSON object.
`{{.IsDeleted}}`	boolean	`true` when Prequel is delivering a record deletion, `false` otherwise.
`{{.Prequel.ProviderID}}`	string	Identifier for the provider whose data is being delivered.
`{{.Prequel.LoadID}}`	string	Identifier for the active load job.
`{{.Prequel.DestinationID}}`	string	Identifier for the destination receiving this delivery.
`{{.Prequel.StreamID}}`	string	Identifier for the stream this delivery came from.

Delivery format

Prequel makes one HTTP request per batch. The request body is an envelope with the following fields:

Field	Type	Description
`url`	string	Presigned URL to the batch file. The URL itself carries a 24-hour TTL set by object storage.
`format`	string	Format of the file at `url`. One of `json`, `csv`, or `parquet`.
`record_count`	integer	Number of records contained in the batch file.
`batch_id`	string	Unique identifier for this batch.
`provider_id`	string	Identifier for the provider whose data is being delivered. Omitted when unavailable.
`load_id`	string	Identifier for the active load job. Omitted when unavailable.

Your endpoint reads the envelope, downloads the file at url, and processes the records inside. The format field on the destination controls what kind of file the URL points to:

Format	File at `url`
`json`	JSONL. Each line is one record, optionally shaped by the `body` template.
`csv`	CSV. Body templates are not supported.
`parquet`	Parquet. Body templates are not supported.

Webhook headers

Prequel includes the following headers on every delivery request.

Header	Description
`X-Prequel-Webhook-Signature`	SHA-256 RSA PKCS1 v1.5 signature of the payload. To verify, fetch the public key from `GET /public/signatures/webhook-public-key` and check the hex-decoded signature against the SHA-256 of `<X-Prequel-Webhook-Timestamp>.<raw body>` using PKCS1 v1.5.
`X-Prequel-Webhook-Timestamp`	RFC 3339 timestamp of when the request was sent. Reject stale timestamps to prevent replay attacks.
`X-Prequel-Webhook-Digest`	SHA-256 hash of the raw request body.
`Content-Type`	Defaults to `application/json` unless overridden by the `headers` template.

Response codes

Your endpoint should respond with an appropriate HTTP status code. Prequel uses the response to determine whether to retry delivery.

Code	Description
`2xx`	Success. Prequel treats a 2xx response as a successful delivery.
`400`	Bad Request. Malformed or missing fields. Prequel will not retry.
`403`	Forbidden. Invalid signature or credentials. Prequel will not retry.
`404`	Not Found. Endpoint or resource could not be found. Prequel will not retry.
`413`	Payload Too Large. Request body exceeds your endpoint’s size limit. Prequel will not retry.
`422`	Unprocessable Entity. Validation failed for specific fields. Prequel will not retry.
`429`	Too Many Requests. Rate limit exceeded. Prequel will back off and retry.
`431`	Request Header Fields Too Large. Prequel will not retry.
Other 4xx	Other client error. Prequel will not retry.
`500`	Internal Server Error. Unexpected error during processing. Prequel will retry.
`502`	Bad Gateway. Upstream error. Prequel will retry.
`503`	Service Unavailable. Prequel will retry.
`504`	Gateway Timeout. Request timed out. Prequel will retry.
Other 5xx	Other server error. Prequel will retry.

Getting Started

Advanced Functionality

Logging and Monitoring

Security and Compliance

Source Configuration Guides

Authentication

Creating your destination spec

Template variables

Delivery format

Webhook headers

Response codes

Getting Started

Advanced Functionality

Logging and Monitoring

Security and Compliance

Source Configuration Guides

Documentation Index

​Authentication

​Creating your destination spec

​Template variables

​Delivery format

​Webhook headers

​Response codes

Authentication

Creating your destination spec

Template variables

Delivery format

Webhook headers

Response codes