Data replication
Backfills
Copy a table's existing rows to your destination while change data capture (CDC) keeps running. Start, cancel and tune backfills from the dashboard.
How it works
A backfill Backfills ensure that all your historical data is copied to your destination. copies a table’s existing rows to your destination. For each table, Artie runs the backfill and change data capture (CDC) side by side:
A table's backfill and CDC
Both start together. Changes wait in Kafka until the last batch lands.
Artie scans the full table in batches and writes them directly to the destination. It can read from a read replica to keep load off your primary database.
Applying the queued changes after the backfill keeps the table consistent: older backfilled rows never overwrite newer changes.
How to kick off a backfill
Artie backfills a new table when you deploy it. Three cases go straight to streaming:
- Events API and S3 sources never backfill a new table.
- DynamoDB sources backfill only when backfill is turned on for the pipeline.
- A table with
skipBackfillset through the API, orskip_backfillin Terraform, skips its first backfill.
A backfill you start by hand on a table that is already deployed is an ad hoc backfill.
Run an ad hoc backfill
- Open the pipeline’s overview page.
- Open the More options menu and select Backfill tables….
- Select the tables to backfill, then select Continue.
- Under Existing rows, choose what happens to the destination tables, then select Continue.
- Under Speed, choose Standard or Performance, then select Continue.
- Under Review, check your choices, then select Start backfill.
The overview page shows a progress card for the backfill. Artie saves the speed you chose as the pipeline’s backfill speed, so later backfills use it too.
Some pipelines show different options:
- History tables: select Backfill history tables… instead. It appears when a table has history mode and Backfill history table turned on.
- Unified tables and DynamoDB sources: Backfill tables… opens a dialog with the same destination table options and no Speed step.
- SQL Server tables unified across databases: Backfill databases… backfills every unified table from the databases you pick.
- Shared source readers: the Speed step is read-only.
- View-only users: the menu has no backfill items.
For unified tables, the dialog also limits a backfill to the schemas or databases you pick. That option requires Keep existing rows.
To start a backfill from code, call Backfill pipeline tables. A workspace starts backfills for at most 2,500 tables per hour.
Cancel a backfill
On the pipeline’s overview page, select Cancel backfill on the progress card, then confirm. The More options menu has the same action as Cancel backfill…, and Cancel history backfill… for history tables.
Rows that the backfill already loaded stay in the destination. With Rebuild and swap, the destination table keeps the rows it had before the backfill.
Destination table behavior
For an ad hoc backfill, the Existing rows step sets what happens to the destination tables before the backfill starts. Keep existing rows is selected by default.
Recommended. Artie replaces matching rows and leaves the rest. Rows deleted at the source stay in your destination.
Empties the destination table and keeps its schema, then reloads it. Queries see partial data until the backfill finishes.
Recreates the table from the source, including its schema. Use this when the schema changed.
Loads the backfill into a new table, then swaps it with the destination table when it finishes. Queries never see the table empty or partially loaded. Snowflake and Databricks only, in early access. See Rebuild and swap backfills.
Four cases limit these options:
- Kafka destinations offer only Keep existing messages. Artie publishes the backfill on top of the messages already in the topic.
- History tables don’t support Rebuild and swap.
- Dynamic tables need Empty tables first or Drop and recreate. Artie rejects the other options when you start the backfill. Drop and recreate empties the changes table instead of dropping it.
- Partitioned tables on PostgreSQL destinations with partition mirroring: Drop and recreate empties the table instead of dropping it, so the partitions and schema stay in place.
Advanced settings
Backfill speed and concurrency are set per pipeline. To change them, open the pipeline, select Edit pipeline, and open the Source tab. The changes apply after you deploy.
When you create a pipeline, set the same values under Backfill speed in the Delivery section of the Review and deploy step.
Backfill settings
Backfill settings sets how hard this pipeline’s backfills read from your database. It has two modes:
- Standard (recommended) puts a light load on the database and is safe for live traffic.
- Performance is faster and adds more load, so it works best on a read replica.
Open Advanced tuning to see the values behind each mode. Standard locks every setting except Batch size. Performance unlocks all of them.
| Setting | Label when you create a pipeline | Standard | Performance default | Allowed values |
|---|---|---|---|---|
| Read method | Read mode | Primary key | Key ranges | Primary key, CTID (PostgreSQL only), Key ranges |
| Parallelism | Queries per table | 1 | 10 | 1 to your workspace limit (20 by default, 50 at most) |
| Pagination | Paging | Client | Client | Client, Server |
| Chunk size | Rows per chunk | Not used | 5,000,000 rows | 100,000 to 20,000,000 rows (100,000,000 with CTID) |
| Batch size | Rows per pull | 5,000 rows | 5,000 rows | 1 to 50,000 rows |
How the settings interact:
- Read method: Primary key reads in key order, one batch at a time. CTID and Key ranges split the table and read the parts in parallel.
- Parallelism and Chunk size: these apply only to CTID and Key ranges.
- Pagination: with Client, Artie pages with
ORDER BYand takes no locks. With Server, your database pages withoutORDER BYand locks the source.
Three cases change this section:
- DynamoDB and Events API sources don’t show Backfill settings.
- A pipeline on a shared source reader shows the section as read-only.
- A table with its own CTID, key range or partition settings keeps them over the pipeline’s settings.
To parallelize a single table’s backfill, see table optimizations.
Max concurrent backfills
In the Advanced settings card of the Source tab, Max concurrent backfills sets how many tables backfill at once. It defaults to your workspace limit, which is 10 by default. Leave it blank to use the workspace limit, or enter a value from 1 to that limit.
When you create a pipeline, the same setting is the Tables at once field under Backfill speed > Custom.
Read replica
PostgreSQL, MySQL, SQL Server, Oracle and CockroachDB sources support backfilling from a read replica. In the source’s connection settings, turn on Use a read replica for backfill?, then enter the Replica host and Replica port.
Artie uses the replica only for backfills. CDC still reads from your primary database.
Backfill order
Queued tables start in alphabetical order by table name, up to the pipeline’s Max concurrent backfills limit. The remaining tables wait their turn. A history table’s backfill counts toward the limit as its own table.
Was this page helpful?
- Need help? Message us on Slack or email [email protected].
- Want a walkthrough? Book a demo.
- LLM? Read llms.txt.