Getting started
Quickstart
Create your first Artie pipeline: connect a source, choose tables, connect a destination, and deploy.
Artie uses change data capture (CDC) to replicate data from your source database to your destination in real time.
Prerequisites
- An Artie account (sign up or book a demo)
- Admin access to your source database
- Credentials for your destination, for example Snowflake, BigQuery, or Redshift
Create your first pipeline
In the Artie dashboard, select New pipeline. The setup checklist opens and walks you through each step below.
1. Connect your source
Choose the source, give Artie network access to it, and check the connection.
- If Where it runs opens, choose the Environment and Data plane, then select Continue.
- If Connect agent opens, select Skip for now. The step is optional.
- On the Choose source step, select a source type or an existing source, then select Continue.
- Complete Network access and Prepare source if the checklist lists them for your source.
- On the Connect and verify step, enter the credentials Artie uses.
- Select Verify source.
When the source verifies, the button changes to Continue.
These pages cover the network options and the setup for each source.
Artie’s fixed IPs and instructions for SSH tunnels and PrivateLink.
Setup steps and service account scripts for each source, starting with PostgreSQL.
2. Choose tables
Turn on replication for each table you want in your destination.
- On the Choose tables step, turn on the replication toggle for each table.
- Select Continue.
A replicated table’s toggle reads Replicating, and Replicate all turns on every table. History mode also writes every change to a separate __history table, with the operation type and timestamp: turn on a table’s history toggle, which then reads History on. To change settings for one table, select Columns and settings on its row.
The table settings page lists every per-table setting.
Destination table alias, column exclusion, merge predicates, and the other per-table settings.
3. Connect your destination
Choose the destination and enter its connection details.
- On the Choose destination step, select a destination type or an existing destination, then select Continue.
- Complete Network access and Prepare destination if the checklist lists them for your destination.
- On the Connect and verify step, enter the credentials Artie uses.
- Select Verify destination.
When the destination verifies, the button changes to Continue.
Each destination has a setup guide.
Connection details and permissions for each destination, starting with Snowflake.
4. Review and deploy
Set any optional settings under Delivery on the Review and deploy step, then deploy. Artie checks the source and destination connections before it deploys.
- On the Review and deploy step, select Validate and deploy.
- In the Deploy pipeline dialog, select Deploy.
When the pipeline has new tables and the source isn’t DynamoDB or an API source, the dialog shows Backfill existing data, which is on by default. When it is off, Artie replicates only changes made after you deploy.
After the deploy starts, Artie opens the pipeline’s overview page.
Optional settings
These settings are under Delivery on the Review and deploy step. To change them on a running pipeline, open the pipeline’s Destination tab and select Advanced settings.
Kafka destinations have none of these settings. S3 destinations have no deletes or dropped-columns setting.
| Setting | Default | What it does |
|---|---|---|
| Updated at (under Extra columns) | On | Adds __artie_updated_at, the time Artie last wrote the row |
| Source updated at (under Extra columns) | Off | Adds __artie_db_updated_at, the source database’s time for the change |
| Dropped columns | Never drops columns | Drop from destination drops any destination column that isn’t in the source, including columns added only in the destination. Artie keeps its own __artie_ columns and your static columns |
| Deletes | Deletes the row in the destination | Keep and mark deleted keeps rows deleted in the source, with __artie_delete set to true |
Verify data is flowing
After you deploy, the pipeline’s overview page shows Waiting for first data. When rows reach the destination, the banner changes to Your first data is flowing. The Analytics portal shows the rows synced for a running pipeline.
Next steps
Monitor a live pipeline from the Analytics portal, and follow source schema changes through schema change notifications.
Rows synced, ingestion lag, and other metrics for each pipeline.
Summaries of table and column changes in your source database.
Was this page helpful?
- Need help? Message us on Slack or email [email protected].
- Want a walkthrough? Book a demo.
- LLM? Read llms.txt.