Pipelines

A pipeline connects a source database to a destination and continuously streams changes. Once created, a pipeline handles initial backfills, schema evolution, and ongoing replication without manual intervention. Each pipeline consists of:

• Source reader - Connects to your source database and captures changes via CDC. A source reader can be shared across multiple pipelines so your database is only read once.
• Destination connector - Writes changes to a specific destination (e.g. Snowflake, BigQuery, S3).

You can monitor pipeline health, throughput, and latency from the analytics portal.

​

Writing to multiple destinations

Artie supports reading from a single source and writing to multiple destinations in parallel - for example, streaming changes to both S3 and Snowflake simultaneously. This is configured by creating multiple pipelines that share the same source reader, so your source database is only read once regardless of how many destinations you have. This feature is currently available through Terraform. To set it up, create artie_source_reader resource and reference it across multiple artie_pipeline resources.

​

Pausing and resuming

You can pause a pipeline at any time from the pipeline overview page. Artie offers two pause modes:

• Pause writing only - Artie stops writing to the destination but continues capturing changes from the source. Events accumulate in Kafka, so no data is lost. If you resume within 14 days, all queued changes are applied in order and no backfill is needed.
• Pause reading & writing - Artie stops both reading from the source and writing to the destination entirely. No new changes are captured while paused.

You can also pause individual tables within a pipeline from the tables tab in the pipeline editor.

​

Message compression

Message compression reduces the number of rows silently skipped due to Kafka’s message size limit by compressing the entire Kafka message value before the size check runs. Without compression, Artie measures the raw (uncompressed) byte size of each message and drops any that exceed the per-message limit (default 2 MB). A large JSON column can push a row over that limit even when the actual information is modest. With message compression enabled, Artie compresses the value first, measures the compressed size, and only skips the row if the compressed payload still exceeds the limit — which is far less common. If you want to be notified when rows are skipped, enable pipeline webhooks.

​

Limitations

• Mutually exclusive with column compression. Message compression and column compression cannot both be active on the same pipeline. Enabling one requires the other to be off.

​

Enabling message compression

1. In the dashboard, open your pipeline and go to Edit → Source tab.
2. Toggle Message compression on.
3. Save the pipeline.

​

Autopilot

Autopilot is a feature that will automatically replicate new tables from your source database. To enable autopilot:

1. Go to the pipeline editor
2. Click on the Destination tab
3. Click on the Advanced settings tab
4. Enable Auto-replicate new tables