> ## Documentation Index
> Fetch the complete documentation index at: https://artie.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Redis Destination Connector: Setup and Configuration

> Configure Redis as a destination in Artie. Artie writes each table's changes to a Redis Stream as JSON entries, with support for TLS, ACL users, SSH tunnels, Redis Cluster, and an optional cap on stream length.

## What you'll need

* Redis 5.0 and later, which added [Redis Streams](https://redis.io/docs/latest/develop/data-types/streams/). To sign in with a username, Redis 6.0 and later, which added [ACLs](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/).
* Network access from Artie to your Redis host and port: a direct connection that allows Artie's IP addresses, or an [SSH tunnel](/docs/connection-options/ssh-tunnel).
* If your Redis uses ACLs, a user that can run `XADD` on Artie's stream keys. You create it in the first step.

## Setup

<Steps>
  <Step title="Create a Redis user for Artie">
    Artie writes to keys that start with `redis:<namespace>:`. The namespace is `public` unless you set one on the pipeline. Replace `public` in the commands below if you use a different namespace.

    <Tabs>
      <Tab title="Standalone">
        ```bash theme={null}
        redis-cli ACL SETUSER artie on '>s3cure-Passw0rd' '~redis:public:*' +@connection +xadd
        ```
      </Tab>

      <Tab title="Redis Cluster">
        The cluster client also reads the slot map, so the user needs `CLUSTER SLOTS`:

        ```bash theme={null}
        redis-cli -p 7000 ACL SETUSER artie on '>s3cure-Passw0rd' '~redis:public:*' +@connection +xadd '+cluster|slots'
        ```

        Redis Cluster doesn't copy ACL users between nodes. Run the command on every node, or create the user with your provider's user management, such as [ElastiCache user groups](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Clusters.RBAC.html).
      </Tab>
    </Tabs>

    If the user is missing a permission, Redis rejects the command and the connection test or the write fails with a `NOPERM` error, for example `NOPERM User artie has no permissions to run the 'cluster|slots' command` or `NOPERM No permissions to access a key`. Add the missing command or key pattern to the user.

    If your Redis only uses a password (`requirepass`), skip this step and leave **Username (optional)** blank in the next step.
  </Step>

  <Step title="Add the Redis connector in Artie">
    Create a destination connector, choose **Redis**, and fill in:

    | Setting | What to enter |
    | - | - |
    | **Host** | The Redis hostname or IP address. For Redis Cluster, use the configuration endpoint. |
    | **Port** | The Redis port. Defaults to `6379`. |
    | **Username (optional)** | The ACL user from the previous step. Leave it blank for password-only Redis. |
    | **Password (optional)** | The user's password, or the `requirepass` password. |
    | **Cluster mode** | Turn this on for Redis Cluster, such as ElastiCache with cluster mode enabled. |
    | **Database number** | The logical database to write to. Defaults to `0`. Hidden when **Cluster mode** is on. |
    | **Max stream length** | Roughly how many entries each table's stream keeps. Defaults to `0`, which keeps every entry. See [Stream length and trimming](#stream-length-and-trimming). |
    | **Enable TLS/SSL** | Turn this on if your Redis requires TLS. Artie uses TLS 1.2 or later and verifies the server certificate against **Host**. |

    Redis Cluster only has database `0`. If **Cluster mode** is on and **Database number** isn't `0`, saving fails with `Redis Cluster only supports database 0`.

    For the connection method, choose a direct connection or an SSH tunnel:

    * **SSH tunnel** works for standalone Redis. With TLS on, Artie still verifies the certificate against **Host**, not the tunnel. Redis Cluster can't use a tunnel, because the cluster tells clients to connect to each node's own address, which the tunnel doesn't forward. Saving a cluster connector with a tunnel fails with `SSH tunnels aren't supported for Redis Cluster`.
    * **PrivateLink** isn't supported for Redis. Saving fails with `PrivateLink isn't supported for Redis`.
  </Step>

  <Step title="Set the namespace on your pipeline">
    In the pipeline's destination settings, set **Namespace (optional)** to group this pipeline's streams under one prefix. It defaults to `public`.

    You can't change the namespace while the pipeline is running. Pause the pipeline first, then rename the existing stream keys to the new prefix, or Artie starts writing to new streams and your consumers miss the old ones.
  </Step>
</Steps>

## Advanced

<Accordion title="Stream keys and entry format">
  Artie writes each table to its own [Redis Stream](https://redis.io/docs/latest/develop/data-types/streams/) with the key `redis:<namespace>:<table>`, for example `redis:public:orders`.

  Each stream entry has one field, `data`, holding the row as a JSON object keyed by column name:

  ```bash theme={null}
  redis-cli XRANGE redis:public:orders - + COUNT 1
  ```

  ```text theme={null}
  1) 1) "1791941136611-0"
     2) 1) "data"
        2) "{\"__artie_delete\":false,\"__artie_operation\":\"c\",\"id\":1042,\"price\":\"19.99\",\"status\":\"pending\"}"
  ```

  * Every entry includes `__artie_operation`: `c` for create, `u` for update, `d` for delete, and `r` for rows from a backfill. Artie always adds this column for Redis, so the **Add 'Artie operation' column** setting doesn't apply.
  * `__artie_delete` is `true` when the row was deleted in the source.
  * Decimal values are JSON strings, such as `"19.99"`, so no precision is lost.
  * Other [system columns](/docs/pipelines/system-columns) appear when you turn them on in the pipeline's advanced settings.

  To read the stream with a consumer group:

  ```bash theme={null}
  redis-cli XGROUP CREATE redis:public:orders analytics 0 MKSTREAM
  redis-cli XREADGROUP GROUP analytics worker-1 COUNT 100 STREAMS redis:public:orders '>'
  ```
</Accordion>

<Accordion title="Changes to the same row within a flush">
  Artie buffers changes and writes them when a [flush rule](/docs/pipelines/flush-rules/overview) triggers. Within one flush, it keeps only the latest change for each primary key. If a row is updated three times between flushes, the stream gets one entry with the final values. If a row is created and then deleted between flushes, the stream gets only the delete.

  Shorter flush rules write intermediate changes more often, but don't guarantee one entry per source change.
</Accordion>

<Accordion title="Delivery guarantees and retries">
  Delivery is at least once. Make your consumers idempotent, for example by upserting on the primary key.

  * If a write to Redis fails, for any reason, Artie retries it up to 15 times, waiting between 1 and 30 seconds between attempts. If every attempt fails, Artie reports a replication error, restarts, and tries again from its last committed position.
  * Artie only commits its position after a write succeeds, so a failed write doesn't lose changes. They're written once Redis accepts writes again.
  * Artie sends each flush's entries to Redis in one pipeline, which isn't a transaction. If a write fails partway through, the retry writes the entries that already landed again, and consumers see them twice.
</Accordion>

<Accordion title="Stream length and trimming">
  **Max stream length** caps each table's stream. Artie passes it to [`XADD`](https://redis.io/docs/latest/commands/xadd/) as `MAXLEN ~`, so Redis trims old entries as Artie adds new ones. Trimming is approximate: Redis only removes whole internal nodes, so a stream can run somewhat over the cap. The cap applies to every stream the connector writes.

  <Warning>Trimmed entries are gone. If a consumer falls more than **Max stream length** entries behind, it never sees the entries Redis trimmed. Set the cap high enough to cover your consumers' worst-case lag.</Warning>

  With the default of `0`, Artie never trims, and each stream grows until you trim it with [`XTRIM`](https://redis.io/docs/latest/commands/xtrim/). If Redis reaches `maxmemory` and can't evict, it rejects writes with an `OOM` error. Artie keeps retrying as described in [Delivery guarantees and retries](#delivery-guarantees-and-retries), and the pipeline makes no progress until you free memory or raise `maxmemory`.
</Accordion>

<Accordion title="Redis Cluster">
  With **Cluster mode** on, Artie connects to the configuration endpoint, reads the cluster's slot map, and writes each stream to the node that owns its key. Each flush for a table writes to a single key, so all of its entries go to one node.

  If **Cluster mode** is off and **Host** points at a cluster, Redis answers writes to keys on other nodes with `MOVED <slot> <host>:<port>`. Retrying doesn't help, so the pipeline reports a replication error that contains `MOVED`. Turn on **Cluster mode** to fix it.
</Accordion>

## Limitations

* **PrivateLink isn't supported.** Saving the connector with a PrivateLink connection fails. Connect directly or through an SSH tunnel.
* **Redis Cluster can't use an SSH tunnel.** Saving fails. Connect to the cluster directly and allow Artie's IP addresses.
* **Redis Cluster writes to database `0` only.** This is a Redis Cluster restriction. Use the namespace to separate pipelines.
* **One stream length cap per connector.** **Max stream length** applies to every table. To use different caps, create a separate Redis connector for those tables.
* **Not every source change becomes an entry.** Changes to the same row within one flush collapse into the latest. See [Changes to the same row within a flush](#changes-to-the-same-row-within-a-flush).

## Frequently asked questions

### Why are entries missing from my stream?

Redis trimmed them, or Artie collapsed them within a flush. Check **Max stream length** against how far your consumers fall behind, and see [Changes to the same row within a flush](#changes-to-the-same-row-within-a-flush).

### Why do my consumers see the same entry twice?

A write failed partway and Artie retried it. Delivery is at least once, so make consumers idempotent. See [Delivery guarantees and retries](#delivery-guarantees-and-retries).

### How do I tell a delete from an update?

Read `__artie_operation`. It's `d` for deletes and `u` for updates, and every entry has it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.