Skip to main content

​
What you’ll need

  • Your Snowflake account identifier
  • A role that can create users, roles and warehouses (SECURITYADMIN and SYSADMIN)
  • Ownership of each table you want to replicate, to turn on change tracking
  • A primary key on each table, either declared in Snowflake or set as a primary key override in Artie

​
Setup

1

Generate a key pair

Snowflake sources sign in with key-pair authentication. Generate an RSA key pair:
rsa_key.pub goes on the Snowflake user in the next step. You’ll paste rsa_key.p8 into Artie at the end.
2

Create a role, service user and warehouse

Replace the RSA_PUBLIC_KEY value with the contents of rsa_key.pub, without the BEGIN and END lines.
setup.sql
3

Grant read access

Grant the role access to the warehouse and to each table you want to replicate:
grants.sql
4

Turn on change tracking

As the table owner, turn on change tracking for each table:
Change tracking only records changes made after it’s turned on. Artie snapshots each table first, so nothing before that is missed.
5

Check the table's retention

Artie reads changes from the table’s Time Travel history, so the table needs a DATA_RETENTION_TIME_IN_DAYS of at least 1. That’s Snowflake’s default.On Enterprise Edition, a longer retention lets Artie catch up after an outage of more than a day:
6

Add the source in Artie

In the pipeline wizard, choose Snowflake as the source and enter your account identifier, the service user (ARTIE_READER), the private key from rsa_key.p8, the warehouse (ARTIE_WH) and the database.

​
Advanced

Artie snapshots each table once, then reads its changes with Snowflake’s CHANGES clause:
  • Every 5 minutes, Artie reads each table’s changes since the last read, ending 10 seconds behind Snowflake’s clock so a transaction that’s still committing isn’t missed.
  • After an outage, Artie catches up in windows of up to one hour and saves its place after each one.
  • Up to 4 tables are read at a time.
Each read runs a query on your warehouse, even when a table hasn’t changed. A small warehouse with AUTO_SUSPEND = 60 keeps that cost low.
Snowflake doesn’t enforce primary keys, so Artie uses the key declared on the table. If a table has no declared key, or the declared key isn’t unique, set a primary key override in the table’s settings.A row whose primary key is NULL stops the sync with an error that names the column. Choose key columns that are never NULL.

​
Frequently asked questions

​
Can I replicate dbt models?

Incremental dbt models work as-is. dbt table models run CREATE OR REPLACE TABLE on every run, which drops the table’s change history, and the new table starts without change tracking, so Artie stops syncing the table after each run. Use an incremental materialization for tables you replicate with Artie.