Skip to main content

​
What you’ll need

  • Redis 5.0 and later, which added Redis Streams. To sign in with a username, Redis 6.0 and later, which added ACLs.
  • Network access from Artie to your Redis host and port: a direct connection that allows Artie’s IP addresses, or an SSH tunnel.
  • If your Redis uses ACLs, a user that can run XADD on Artie’s stream keys. You create it in the first step.

​
Setup

1

Create a Redis user for Artie

Artie writes to keys that start with redis:<namespace>:. The namespace is public unless you set one on the pipeline. Replace public in the commands below if you use a different namespace.
If the user is missing a permission, Redis rejects the command and the connection test or the write fails with a NOPERM error, for example NOPERM User artie has no permissions to run the 'cluster|slots' command or NOPERM No permissions to access a key. Add the missing command or key pattern to the user.If your Redis only uses a password (requirepass), skip this step and leave Username (optional) blank in the next step.
2

Add the Redis connector in Artie

Create a destination connector, choose Redis, and fill in:Redis Cluster only has database 0. If Cluster mode is on and Database number isn’t 0, saving fails with Redis Cluster only supports database 0.For the connection method, choose a direct connection or an SSH tunnel:
  • SSH tunnel works for standalone Redis. With TLS on, Artie still verifies the certificate against Host, not the tunnel. Redis Cluster can’t use a tunnel, because the cluster tells clients to connect to each node’s own address, which the tunnel doesn’t forward. Saving a cluster connector with a tunnel fails with SSH tunnels aren't supported for Redis Cluster.
  • PrivateLink isn’t supported for Redis. Saving fails with PrivateLink isn't supported for Redis.
3

Set the namespace on your pipeline

In the pipeline’s destination settings, set Namespace (optional) to group this pipeline’s streams under one prefix. It defaults to public.You can’t change the namespace while the pipeline is running. Pause the pipeline first, then rename the existing stream keys to the new prefix, or Artie starts writing to new streams and your consumers miss the old ones.

​
Advanced

Artie writes each table to its own Redis Stream with the key redis:<namespace>:<table>, for example redis:public:orders.Each stream entry has one field, data, holding the row as a JSON object keyed by column name:
  • Every entry includes __artie_operation: c for create, u for update, d for delete, and r for rows from a backfill. Artie always adds this column for Redis, so the Add ‘Artie operation’ column setting doesn’t apply.
  • __artie_delete is true when the row was deleted in the source.
  • Decimal values are JSON strings, such as "19.99", so no precision is lost.
  • Other system columns appear when you turn them on in the pipeline’s advanced settings.
To read the stream with a consumer group:
Artie buffers changes and writes them when a flush rule triggers. Within one flush, it keeps only the latest change for each primary key. If a row is updated three times between flushes, the stream gets one entry with the final values. If a row is created and then deleted between flushes, the stream gets only the delete.Shorter flush rules write intermediate changes more often, but don’t guarantee one entry per source change.
Delivery is at least once. Make your consumers idempotent, for example by upserting on the primary key.
  • If a write to Redis fails, for any reason, Artie retries it up to 15 times, waiting between 1 and 30 seconds between attempts. If every attempt fails, Artie reports a replication error, restarts, and tries again from its last committed position.
  • Artie only commits its position after a write succeeds, so a failed write doesn’t lose changes. They’re written once Redis accepts writes again.
  • Artie sends each flush’s entries to Redis in one pipeline, which isn’t a transaction. If a write fails partway through, the retry writes the entries that already landed again, and consumers see them twice.
Max stream length caps each table’s stream. Artie passes it to XADD as MAXLEN ~, so Redis trims old entries as Artie adds new ones. Trimming is approximate: Redis only removes whole internal nodes, so a stream can run somewhat over the cap. The cap applies to every stream the connector writes.
Trimmed entries are gone. If a consumer falls more than Max stream length entries behind, it never sees the entries Redis trimmed. Set the cap high enough to cover your consumers’ worst-case lag.
With the default of 0, Artie never trims, and each stream grows until you trim it with XTRIM. If Redis reaches maxmemory and can’t evict, it rejects writes with an OOM error. Artie keeps retrying as described in Delivery guarantees and retries, and the pipeline makes no progress until you free memory or raise maxmemory.
With Cluster mode on, Artie connects to the configuration endpoint, reads the cluster’s slot map, and writes each stream to the node that owns its key. Each flush for a table writes to a single key, so all of its entries go to one node.If Cluster mode is off and Host points at a cluster, Redis answers writes to keys on other nodes with MOVED <slot> <host>:<port>. Retrying doesn’t help, so the pipeline reports a replication error that contains MOVED. Turn on Cluster mode to fix it.

​
Limitations

  • PrivateLink isn’t supported. Saving the connector with a PrivateLink connection fails. Connect directly or through an SSH tunnel.
  • Redis Cluster can’t use an SSH tunnel. Saving fails. Connect to the cluster directly and allow Artie’s IP addresses.
  • Redis Cluster writes to database 0 only. This is a Redis Cluster restriction. Use the namespace to separate pipelines.
  • One stream length cap per connector. Max stream length applies to every table. To use different caps, create a separate Redis connector for those tables.
  • Not every source change becomes an entry. Changes to the same row within one flush collapse into the latest. See Changes to the same row within a flush.

​
Frequently asked questions

​
Why are entries missing from my stream?

Redis trimmed them, or Artie collapsed them within a flush. Check Max stream length against how far your consumers fall behind, and see Changes to the same row within a flush.

​
Why do my consumers see the same entry twice?

A write failed partway and Artie retried it. Delivery is at least once, so make consumers idempotent. See Delivery guarantees and retries.

​
How do I tell a delete from an update?

Read __artie_operation. It’s d for deletes and u for updates, and every entry has it.