# Single Sign-On (SSO) Source: https://artie.com/docs/account/enabling-okta Customize how your team can log into Artie Artie supports single sign-on to your dashboard via OIDC. If you are using Okta, we have provider specific documentation for setting up SSO. If you are using another provide, just contact us! You need to be your company's Okta and Artie admin to set this up To create an app, please do the following: 1. Log in to your Okta admin dashboard 2. Go to Applications > Applications 3. Click "Create App Integration", select "OIDC - OpenID Connect", and pick "Web Application" * Application Name: Artie * Grant type: Only "Authorization Code" should be selected * Sign-in redirect URIs: `https://api.artie.com/sso/okta/callback` * Sign-out redirect URIs: `https://api.artie.com/logout` * (Optional) If you want to enable Artie as an Okta tile app. You can also specify `https://api.artie.com/sso/okta/login` as the "Initiate login URI" * Client ID * Client secret * Okta domain (You can get this from the top right navigation bar) ![Client ID and secret](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/okta/client_id_secret.png) ![Retrieving your Okta domain](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/okta/okta_url.png) # Configuring this on Artie Once you have gathered the information from above, go to the [settings](\(https://app.artie.com/settings\)) page in your Artie dashboard. Under "Authentication", you can click "Configure Okta SSO". Once all the necessary information is filled out, users can now [log in](https://app.artie.com/login/sso) through Okta SSO. ## Questions To log in through SSO, users would: 1. Go to the [SSO login screen](https://app.artie.com/login/sso) 2. Enter their company email address 3. If matched, an email will be sent to the user to log in 4. It will prompt them to log in through their configured IDP 5. They will then be redirected and authenticated into Artie # Enabling Slack Notifications Source: https://artie.com/docs/account/enabling-slack-notifications Steps to get alerts in your Slack channel: 1. Open your Slack and find the channel you want alerts to be sent to 2. Click on `View channel details` > `Integrations` > `Send emails to this channel` ![Generate an email address for your Slack channel](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/slack_email.png) 3. In Artie, go to [Settings > Advanced settings](https://app.artie.com/settings?tab=advanced). 4. Enter the email address under `Additional alert email addresses` and click `Save`. # Inviting Your Team Source: https://artie.com/docs/account/inviting-your-team Once you log in, the first thing you want to do is invite your team. 1. Go to [Settings > Team members](https://app.artie.com/settings?tab=members) and click `+ Invite a team member`. 2. Enter the name and work email address of the person you would like to invite. 3. They will receive an email with a link to sign up for Artie and join your company's account. # Architecture Source: https://artie.com/docs/concepts/architecture Learn about Artie's split plane architecture that separates control and data planes for enhanced security and flexibility. # Overview Deployment model for Artie Artie follows a split plane architecture, where the control and data plane are separated. All data is processed in the data plane and can be deployed in your VPC. ## Deployment Options ### Cloud Deployment * Data processing occurs in Artie's cloud infrastructure * Data plane is co-located with your infrastructure to minimize network costs * Fully managed service with minimal operational overhead ### On-Premise Deployment * All data processing happens within your network * Secure HTTPS endpoint for essential operations: * Basic validation * Metadata retrieval * Control plane communication * Complete data control with local processing # Backfills Source: https://artie.com/docs/concepts/backfill Going over how backfills work in Artie ## Overview The following circumstances may trigger a backfill: When you launch a new Artie deployment, we will backfill all the tables. We will backfill the new tables you have added. Trigger a backfill for a specific table on demand from the deployment overview page. ## How it works You can specify a read replica for backfills to minimize any impact on your primary database. Artie backfills by having parallel processes per table (default is 5). This speeds up the copying process without adding too much load to your database. While backfilling, Artie will continue to read your database logs, capturing all new changes in real-time into our Kafka topics. Once the backfill is complete, Artie process will then switch to consume from the Kafka topics. # Connection Options Source: https://artie.com/docs/connection-options Discover the various ways that Artie is able to connect to your database. To enable Artie to connect to your databases, you have a few options: Add our fixed IPs to your allowlist. If we cannot connect directly to the database, we can connect via an SSH tunnel. Private connectivity between VPCs and supported AWS services. ## Options Artie leverages a split plane architecture. * Our control plane houses our API server. * Our data plane performs the actual data replication. If you plan to safelist Artie's IPs, please ensure you include both our control plane and data plane IPs. #### Control Plane IPs (CIDR format) ``` 52.55.119.205/32 54.204.194.211/32 52.200.253.111/32 52.44.73.206/32 44.217.73.84/32 ``` #### Data Plane IPs (CIDR format) **AWS US-East-1 (Default)** ``` 3.215.55.30/32 3.216.86.119/32 3.234.42.107/32 3.209.157.160/32 3.212.233.181/32 3.216.214.184/32 34.225.190.48/32 54.165.47.156/32 44.216.116.232/32 54.164.238.160/32 ``` **AWS US-West-2** ``` 54.191.160.248/32 52.35.174.184/32 54.71.53.235/32 44.238.111.7/32 100.20.16.250/32 54.245.136.219/32 52.33.116.144/32 44.238.174.20/32 54.70.103.212/32 54.185.25.39/32 ``` 1. Create a new EC2 instance (or equivalent). 2. Record the public IP address and port. 3. Go to your [company settings page](https://app.artie.com/settings) and click `New SSH tunnel`. 4. Copy the public IP address and port from step 2 and click save. 5. Copy the public key and add this to `~/.ssh/authorized_keys` on your SSH tunnel ```bash # (Optional) if you want to create a service account sudo adduser -m artie sudo su artie mkdir ~/.ssh # Add the public key to ~/.ssh/authorized_keys vi ~/.ssh/authorized_keys chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys ``` We support connections through AWS PrivateLink. In order to set this up, please reach out to our team ([hi@artie.com](mailto:hi@artie.com)). 1. You will need to create a VPC service that maps to your RDS instance. 2. Please safelist Artie's principal `arn:aws:iam::788826912768:root` 3. Once this is done, please reach out to our team with your service name we will initiate a VPC endpoint request. ### Questions To find out, please visit your [company's settings page](https://app.artie.com/settings) and see `Data processing location` ![Data processing location](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/data_processing_location.png) Get in touch with us either through our [Slack](https://artie.com/slack) or [email](mailto:hi@artie.com). We are happy to set up another region. We're sorry! We were unable to secure a CIDR range when we first launched. We have now worked this out with AWS and will be using larger CIDR ranges for the next data plane. # BigQuery Source: https://artie.com/docs/destinations/bigquery ## Required settings * Project identifier * Dataset * Service account Go into your [BigQuery console](https://console.cloud.google.com/bigquery) and the project ID is at the top of the navigation bar. ![GCP project id](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/gcp_project_id.png) To find this out, go to [BigQuery console](https://console.cloud.google.com/bigquery), and you will see the dataset name on the left side of the screen. ![BigQuery dataset](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/bigquery_dataset.png) To create a service account, please do the following: 1. Go to the `IAM & Admin` section in your GCP console and click `Service Accounts`. 2. Click on `Create Service Account`, give it a name and click `Create and Continue`. 3. Grant the `BigQuery Admin` role to the service account click `Done`. 4. Go into the service account you just created and click on `Keys` and then `Add Key`. 5. Choose `JSON` and click `Create`. This will download a JSON file with your credentials. ### Best practices Consider [partitioning](/guides/database/enable-bigquery-partitioning) your large tables to improve performance and reduce costs. # Databricks Source: https://artie.com/docs/destinations/databricks ## Required settings * SQL warehouse host and path * Unity catalog name * Volume name * Personal access token ### Volume Artie will write our temporary files to a volume in the catalog and schema that we are syncing into. ```sql CREATE VOLUME my_catalog.my_schema.my_volume; ``` Go into your Databricks workspace, select the SQL warehouse you want to use and click on Connection details. From there, you will be able to find the host and path of your SQL warehouse. ![Databricks connection details](https://mintlify.s3.us-west-1.amazonaws.com/artie/destinations/databricks.png) To generate a personal access token, go into your workspace and do the following: 1. Click on your username in the top bar and select `Settings` from the dropdown. 2. Click `Developer` 3. Click `Access tokens` then `Generate new token` 4. (Optional) Provide a description for the token and if you set expiration, make sure to note it down. # Microsoft SQL Server Source: https://artie.com/docs/destinations/microsoft-sql-server ## Required settings * Host name * Port (default is `1433`) * Service account * Database name * Schema name (default is `dbo`) ### Creating a service account ```sql -- Create a service account CREATE LOGIN artie_transfer WITH PASSWORD = 'AStrongPassword!'; GO USE database; GO CREATE USER artie_transfer FOR LOGIN artie_transfer; GO -- If it's not dbo, change the schema name GRANT ALTER ON SCHEMA::dbo TO artie_transfer; GO GRANT CREATE TABLE, INSERT, UPDATE, DELETE TO artie_transfer; GO ``` # Redshift Source: https://artie.com/docs/destinations/redshift ## Required settings * Redshift endpoint * Schema name * Service account ![Redshift endpoint](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/redshift_cluster_endpoint.png) ### Creating a service account ```sql CREATE USER artie_transfer WITH PASSWORD 'password'; GRANT SELECT, INSERT, UPDATE, DELETE, DROP, ALTER ON ALL TABLES IN SCHEMA schema_name TO artie_transfer; GRANT CREATE, USAGE ON SCHEMA schema_name TO artie_transfer; ``` # S3 Source: https://artie.com/docs/destinations/s3 Artie will write delta files in Parquet format to an S3 bucket. ## Settings * Bucket (required) * Folder name (optional) * Service account (required) ### Bucket Structure Artie will write all the delta files into an S3 bucket. The bucket structure will be as follows: ``` -- tableName will be the fully qualified table name (db.schema.tableName) /{{bucketName}}/{{folderName}}/{{tableName}}/{{YYYY-MM-DD}}/{{unixTimestampMs}}_{{randomString(4)}}.parquet.gz -- example /artie/foo/db.schema.tableName/2023-08-06/1654320000000_abc1.parquet.gz ``` ```hcl provider "aws" { region = "us-east-1" # Your AWS Region } resource "aws_iam_user" "artie_transfer" { name = "artie-transfer" # Your service account name } resource "aws_iam_policy" "s3_read_write_policy" { name = "s3-read-write-policy" description = "Ability to list, read and write to the bucket" policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Action = [ "s3:ListBucket" ] Resource = "arn:aws:s3:::your-bucket-name" }, { Effect = "Allow" Action = [ "s3:GetObject", "s3:PutObject" ] Resource = "arn:aws:s3:::your-bucket-name/*" } ] }) } resource "aws_iam_user_policy_attachment" "user_policy_attachment" { user = aws_iam_user.artie_transfer.name policy_arn = aws_iam_policy.s3_read_write_policy.arn } resource "aws_iam_access_key" "artie_transfer_key" { user = aws_iam_user.artie_transfer.name } output "access_key_id" { value = aws_iam_access_key.artie_transfer_key.id sensitive = false } output "secret_access_key" { value = aws_iam_access_key.artie_transfer_key.secret sensitive = true } ``` # Snowflake Source: https://artie.com/docs/destinations/snowflake ## Required settings * Account URL * Virtual warehouse * Service account * Database name * Schema name You can find your account URL at the bottom left corner of your Snowflake console. ![Retrieving Account URL](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/snowflake_account_url.gif) ```sql BEGIN TRANSACTION; USE ROLE ACCOUNTADMIN; -- This combines both SYSADMIN and SECURITYADMIN -- IMPORTANT, PLEASE FILL THIS OUT AND SAVE THIS -- SET ROLE_NAME = 'ARTIE_TRANSFER_ROLE'; SET SERVICE_USER = 'ARTIE'; SET SERVICE_PW = 'PASSWORD'; SET DWH_NAME = UPPER('DWH'); SET DB_NAME = UPPER('DB_NAME'); SET SCHEMA_NAME = UPPER('public'); -- END IMPORTANT -- SET DB_SCHEMA_NAME = CONCAT($DB_NAME, '.', $SCHEMA_NAME); CREATE ROLE IF NOT EXISTS identifier($ROLE_NAME); CREATE USER IF NOT EXISTS identifier($SERVICE_USER) password = $SERVICE_PW default_role = $ROLE_NAME; GRANT ROLE identifier($role_name) to USER identifier($SERVICE_USER); CREATE WAREHOUSE IF NOT EXISTS identifier($DWH_NAME) warehouse_size = xsmall warehouse_type = standard auto_suspend = 10 auto_resume = true initially_suspended = true; CREATE DATABASE IF NOT EXISTS identifier($DB_NAME); CREATE SCHEMA IF NOT EXISTS identifier($DB_SCHEMA_NAME); GRANT USAGE ON WAREHOUSE identifier($DWH_NAME) TO ROLE identifier($ROLE_NAME); GRANT USAGE ON DATABASE identifier($DB_NAME) TO ROLE identifier($ROLE_NAME); GRANT ALL PRIVILEGES ON SCHEMA identifier($DB_SCHEMA_NAME) TO ROLE IDENTIFIER($ROLE_NAME); GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA identifier($DB_SCHEMA_NAME) TO ROLE IDENTIFIER($ROLE_NAME); GRANT ALL PRIVILEGES ON FUTURE TABLES IN SCHEMA identifier($DB_SCHEMA_NAME) TO ROLE IDENTIFIER($ROLE_NAME); COMMIT; ``` To use key-pair authentication, you will want to generate an RSA key pair. ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub ``` Then upload the public key to Snowflake ```sql ALTER USER ARTIE SET RSA_PUBLIC_KEY='MIIBIjANBgkqh...'; ``` Once this is done, provide us the private key to authenticate with. ## Snowflake eco mode Snowflake eco mode is a great option if you only need faster data syncs during business hours. Click [here](https://www.artie.com/blogs/snowflake-eco-mode) to learn more! 1. Edit your deployment 2. Go into the `Destinations` tab 3. Open `Advanced settings` and select `+ New schedule` 4. Define your peak hours and your time zone 5. Define your flush rules for off-peak hours 6. Save schedule 7. Deploy your changes ![Setting up Snowflake eco mode](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/snowflake_eco_mode.png) ## Troubleshooting To start, We would recommend you to go with a `x-small` and decide whether or not you need to scale based off of our Snowflake insights metrics from our Analytics portal. Generally speaking, if your overall query execution time is high, then you may want to scale to a larger instance. If your queued overload time is high, you may want to have more virtual warehouse nodes to increase your parallelizable slots. Snowflake's native RBAC makes it so that the account that created the resource is the native owner. To change this, assign the ARTIE service account's role to your account and you will be able to operate on the table. See the GIF below on how to fix this problem. ![Snowflake access control](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/SFLK.gif) # Artie's Typing Library Source: https://artie.com/docs/guides/artie/arties-typing-library Learn how Artie's internal typing library ensures data integrity and schema evolution across your data sources. ## Overview 🎯 Artie's custom [typing library](https://github.com/artie-labs/transfer/tree/master/lib/typing) is the backbone of our data transfer system. It ensures that your data maintains its original structure and types when moving between sources and destinations. ### Key Features ✨ * **Schema Evolution**: Automatically detects and adds missing columns * **Type Inference**: Intelligently determines data types from values * **Performance**: 2x faster than Go's Reflect library * **Data Integrity**: Zero data transformation or loss ## Data Type Handling 🔍 Artie's typing library intelligently handles data type mapping between your source and destination systems. Here's how it works: 1. **Initial Type Detection**: When data first arrives, Artie analyzes the source schema and values to determine the most appropriate data type for your destination. 2. **Type Preservation**: Once a data type is established in your destination, Artie maintains that type to ensure consistency. 3. **Automatic Conversion**: If source and destination types differ, Artie automatically converts the data while preserving its integrity. If Artie encounters a type conversion that could lead to data loss or errors, we'll: * Raise a clear error message * Notify you immediately * Work with you to resolve the issue ### Numbers We preserve the exact numeric type from your source (Postgres, MySQL, or MongoDB). Here's how we handle different number formats: * `5` → Integer * `"5"` → String * `5.0` → Float ### JSON Objects We maintain JSON objects exactly as they appear in your source, with no flattening or transformation. > 💡 **Pro Tip**: If you prefer to store JSON as a string, you can explicitly specify the column type as `STRING` in your schema. ### Arrays We support both simple and complex array structures: * Simple arrays * Nested arrays with objects ![Normal array](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/snowflake_array.png) ![Array with nested objects](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/snowflake_array_nested_objects.png) ### Timestamps, Dates, and Times ⏰ Our library supports 15+ different time formats with zero precision loss. We preserve your original time layout using our custom `time.Time` object. > 💡 **Pro Tip**: Like JSON objects, you can override time type inference by specifying your preferred data type in the schema. ## Need Help? 🤝 If you have questions not covered here, reach out to us at [hi@artie.com](mailto:hi@artie.com)! # Column Inclusion and Exclusion Source: https://artie.com/docs/guides/artie/column-inclusion-exclusion Learn how column inclusion and exclusion works in Artie. ## Overview | Feature | Description | | ---------------- | ------------------------------------------------- | | Column Inclusion | Replicate only the columns you specify | | Column Exclusion | Replicate all columns except the ones you exclude | ![Column Inclusion and Exclusion](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/columns/column_decision.png) ## Advanced use case: One source to multiple destinations Artie allows you to replicate data from a single source reader to multiple pipeline tables. To do this, you can do it through our [Terraform provider](https://registry.terraform.io/providers/artie-labs/artie/latest). **Architecture** ```mermaid graph LR A[Source Reader] --> C[Pipeline Table 1] A[Source Reader] --> D[Pipeline Table 2] ``` Each pipeline table will have its own Kafka consumer group and we can specify additional column inclusion and exclusion for each pipeline table. You can specify column inclusion and exclusion at the source reader and pipeline table level, please see the diagram below for guidance on where to set the column settings. ![Column Inclusion and Exclusion](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/columns/source_reader_pipeline_table.png) # Schema Evolution Source: https://artie.com/docs/guides/artie/schema-evolution Learn how Artie automatically evolves schemas to match your data. ## Overview Artie supports schema evolution out of the box, which means: * We automatically **create** tables in your destination if they don't exist * We detect and **add new columns** automatically * We can drop deleted columns from the source (optional feature) ### How do we know the right data types? 🔍 * We automatically detect data types from your source schema and values * For more details, see our [typing library documentation](/guides/artie/arties-typing-library#data-type-handling-%F0%9F%94%8D) You can enable this feature by setting `Enable hard deletes for columns` to `true` in your destination's advanced settings. ![Enable hard deletes for columns](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/hard_delete_columns.png) For safety reasons, columns are not automatically dropped from destination tables since this operation is nearly irreversible. While some destinations support undos, we implement a strict safety check process: 1. When a column is deleted from the source, the first row that omits this column marks it for potential removal 2. We then monitor CDC events for 6 hours to ensure the column is consistently omitted 3. Only after this verification period will we proceed with dropping the column This safety mechanism helps prevent accidental data loss and ensures the column is truly no longer needed. ### Advanced Features 🛠️ We provide additional configuration options to give you more control over schema evolution: Use column exclusion to prevent specific columns from being added to your destination. This is particularly useful for: * Excluding sensitive data columns (e.g., PII, passwords) * Preventing unnecessary columns from being synced * Managing storage costs by excluding large or unused columns To use this feature, simply specify the columns you want to exclude in your table's advanced settings. Column inclusion allows you to explicitly define which columns should be synced to your destination. This feature: * Gives you complete control over your destination schema * Helps maintain a clean, minimal dataset * Requires manual management of new columns Column inclusion is mutually exclusive with column exclusion. When using this feature, you'll need to manually add any new columns to your destination schema as they appear in the source. # Database Migrations Source: https://artie.com/docs/guides/database/database-migrations Learn how to safely perform database migrations while maintaining data replication with Artie. ## Overview 🎯 Database migrations are a critical part of maintaining and upgrading your database infrastructure. This guide explains how to safely perform database migrations while ensuring your data replication continues to work correctly with Artie. ## Major version upgrade ⚠️ Major version upgrades (e.g., PostgreSQL 14 to 15) require stopping database replication to perform the necessary upgrade and restart your database. This is because major version upgrades often include breaking changes that can affect replication. **Steps to perform major version upgrades without data loss:** If you are using Postgres, make sure to drop the replication slot after you paused the Artie deployment. See the [Postgres tips](#postgres-tips) section for more information. 1. Schedule an application downtime and pause your application. 2. Check that Artie is no longer processing any data from our Analytics portal and pause your Artie deployment. 3. Perform the database upgrade and restart your database server. 4. Resume your Artie deployment. 5. Resume your application. ### Database-specific considerations * Ensure `wal_level` is set to `logical` in your new version * Verify that all required extensions are available in the new version * Check that your replication slots are properly configured * Verify `binlog_format` is set to `ROW` * Ensure `binlog_retention_hours` is set appropriately (recommended 24+ hours) * Check that GTID is enabled if you're using it * Ensure your replica set is properly configured in the new version * Verify that change streams are enabled * Check that your service account has the correct permissions ## Minor version upgrade Minor version upgrades (e.g., PostgreSQL 14.1 to 14.2) do not require stopping database replication. These upgrades are typically handled automatically by your database provider and applied during your maintenance window. While minor version upgrades are generally safe, it's still recommended to: * Monitor your replication lag during the upgrade * Have a rollback plan ready * Test the upgrade in a staging environment first ## Best practices ⭐ 1. Always test migrations in a staging environment first 2. Back up your data before any major version upgrade 3. Monitor replication lag during and after the upgrade 4. Keep your Artie deployment paused until the upgrade is complete 5. Verify data consistency after the upgrade ## Troubleshooting 🔧 For Postgres, Artie will find or create a replication slot (default name is `artie`, unless you specified an override) and we will start capturing changes moving forward. For other data sources, Artie will seek the earliest checkpoint from the database and start replicating from that point. If you experience data loss, please reach out to us and we will kick off a backfill process. We recommend: 1. Document the extent of data loss 2. Note the approximate time window affected 3. Contact our support team immediately 4. Keep your deployment paused until we can assist For Postgres, you can run this query to see all the replication slots on the server: ```sql SELECT slot_name, wal_status, pg_size_pretty( pg_wal_lsn_diff( pg_current_wal_lsn(), restart_lsn)) AS retained_wal, active, restart_lsn FROM pg_replication_slots; ``` Ensure that `active` is false and then drop the replication slot. ```sql SELECT pg_drop_replication_slot('artie'); ``` Dropping replication slots will delete any unprocessed WAL files. Make sure you have paused your Artie deployment before dropping slots. 1. **Replication slot overflow**: If you see this error, you may need to increase your WAL retention period 2. **Connection issues**: Verify your database credentials and network connectivity 3. **Permission errors**: Ensure your service account has the correct permissions after the upgrade 4. **Schema changes**: Check if any schema changes were made during the upgrade that might affect replication # Enable BigQuery Partitioning Source: https://artie.com/docs/guides/database/enable-bigquery-partitioning Enable partitioning to lower your merge latency and reduce the amount of bytes scanned. ## Steps to turn on partitioning For this example, consider this table in Postgres. ```sql CREATE TABLE events ( event_id SERIAL PRIMARY KEY, event_name VARCHAR(255) NOT NULL, description TEXT, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); ``` 1. First pause your Artie deployment 2. Recreate the table in BigQuery and make sure to use the right partitioning strategy. ```sql -- Recreate the partitioned table CREATE TABLE artie.events_copy PARTITION BY DATE(created_at) AS SELECT * FROM artie.events; -- Drop the old table DROP TABLE artie.events; -- Rename new to old ALTER TABLE artie.events_copy RENAME TO events; ``` 3. Edit your deployment and update the table settings for `events` 4. Click `Save` and `Deploy` A partitioned table is divided into segments, called partitions, that make it easier to manage and query your data. By dividing a large table into smaller partitions, you can improve query performance and control costs by reducing the number of bytes read by a query. You partition tables by specifying a partition column which is used to segment the table. * Improve query performance by scanning a partition. * When you exceed the [standard table quota](https://cloud.google.com/bigquery/quotas#standard_tables). * Gain partition-level management features such as writing to or deleting partition(s) within a table. * Reduce the number of bytes processed + reduce your BigQuery bill | Partitioning type | Description | Example | | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Time partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#date_timestamp_partitioned_tables) |

Partitioning a particular column that is a TIMESTAMP.

BigQuery allows hourly, daily, monthly, yearly and integer range partitioning intervals.

|

Column: timestamp
Partitioning granularity: daily

| | [Integer range or interval based partitions](https://cloud.google.com/bigquery/docs/partitioned-tables#integer_range) | Partitioning off of a range of values for a given column. |

Say you have a column called customer\_id and there are 100 values.

You can specify to have values 0-9 go to one partition, 10-19 the next, etc.

| | [Ingestion-based](https://cloud.google.com/bigquery/docs/partitioned-tables#ingestion_time) |

This is when the row was inserted.

This is not recommended, because it requires storing additional metadata to know when this row was inserted. If we don't specify this upon a merge, we will end up creating duplicate copies.

| NA | # Preventing WAL Growth on RDS Postgres Source: https://artie.com/docs/guides/database/preventing-postgres-wal-growth-on-rds Learn how to prevent Write-Ahead Log (WAL) growth issues in your AWS RDS Postgres database, especially for low-traffic or test environments. Last updated: 03/26/2025 Write-Ahead Logging (WAL) is Postgres's built-in mechanism for ensuring data integrity and enabling change data capture (CDC). Here's how it works: 1. Every database change is first written to the WAL 2. The changes are then applied to the actual database files 3. This approach ensures data durability and enables reliable replication While WAL is essential for data integrity and replication, uncontrolled WAL growth can lead to: * Replication slot overflow * Database storage exhaustion * Potential database downtime **TL;DR:** 🚨 AWS RDS uses internal "heartbeats" that generate WAL entries every 5 minutes, which can cause significant storage issues on idle databases. **Detailed Explanation:** As explained by Gunnar Morling in his [blog post](https://www.morling.dev/blog/insatiable-postgres-replication-slot/), AWS RDS writes a heartbeat to an internal `rdsadmin` table every 5 minutes. Here's why this matters: * Each WAL segment is 64MB by default * Each heartbeat creates a new WAL segment * On idle databases, this means: * 64MB of WAL growth every 5 minutes * \~18.4GB of WAL growth per day * Potential replication slot overflow if left unchecked This is particularly problematic for: * Test databases * Low-traffic environments * Idle databases The issue doesn't affect active databases because their WAL is constantly being drained by regular data changes. ## 🛡️ Preventing WAL Growth with Heartbeats This solution is specifically designed for low-traffic or idle databases. Active databases don't need this feature as their WAL naturally resets with regular data changes. ### Setup Steps 1. Create and configure the heartbeat table: ```sql -- Create the heartbeat table CREATE TABLE test_heartbeat_table (id text PRIMARY KEY, ts timestamp); -- Grant necessary permissions GRANT UPDATE ON TABLE test_heartbeat_table TO artie_transfer; -- Insert initial record INSERT INTO test_heartbeat_table (id, ts) VALUES (1, NOW()); ``` 2. Enable heartbeats in your deployment's advanced settings. ### Troubleshooting Guide If you're still experiencing WAL growth after enabling heartbeats, check these common issues: Verify `test_heartbeat_table` exists in your database ```sql -- Check if the table is included in your publication -- For Artie deployments, the publication name should default -- to `dbz_publication` (unless changed under Deployment advanced settings) SELECT pubname, tablename FROM pg_publication_tables WHERE tablename = 'test_heartbeat_table'; ``` Confirm the service account has proper write permissions ```sql -- Check for queries that might block replication SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state FROM pg_stat_activity WHERE (now() - pg_stat_activity.query_start) > interval '1 minute'; ``` To further protect your RDS instance, implement these measures: 1. **Monitoring** * Set up alerts for `free_storage_space` * Monitor WAL growth rates 2. **Database Configuration** * Set appropriate `statement_timeout` * Configure `max_slot_wal_keep_size` (default is -1 for unlimited) * Enable storage autoscaling ([AWS Guide](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.ModifyingAutoscaling)) ```sql -- View all replication slots SELECT * FROM pg_replication_slots; -- Remove a replication slot SELECT pg_drop_replication_slot('REPLICATION_SLOT_NAME'); -- Check replication slot size SELECT slot_name, wal_status, pg_size_pretty( pg_wal_lsn_diff( pg_current_wal_lsn(), restart_lsn)) AS retained_wal, active, restart_lsn FROM pg_replication_slots; ``` # Tables Without Primary Keys Source: https://artie.com/docs/guides/database/tables-without-primary-key-s Learn how to add primary keys to your tables and why they're essential for data replication. ## Why Primary Keys Matter 🔑 Primary keys are crucial for data replication because they: 1. **Ensure Data Ordering**: We use primary keys as partition keys in Kafka to guarantee the correct sequence of operations 2. **Maintain Data Integrity**: Primary keys enable reliable `MERGE` operations to keep your data consistent ## Adding Primary Keys: Two Common Scenarios 🛠️ Let's say you have a `users_no_pk` table with a unique email field: ```sql CREATE TABLE users_no_pk ( email VARCHAR(255) UNIQUE NOT NULL, first_name VARCHAR(255) NOT NULL, last_name VARCHAR(255) NOT NULL ); ``` To make this table replication-ready, simply promote the unique email to a primary key: ```sql ALTER TABLE users_no_pk ADD PRIMARY KEY (email); ``` For tables without any unique identifiers, we can add a new primary key column. Let's use this example: ```sql CREATE TABLE no_primary_keys ( key VARCHAR(5), value bool ); INSERT INTO no_primary_keys (key, value) VALUES ('foo', true), ('bar', false); ``` Current table contents: ```bash postgres=# select * from no_primary_keys; key | value -----+------ bar | f foo | f (2 rows) ``` Add a new primary key column: ```sql ALTER TABLE no_primary_keys ADD COLUMN pk SERIAL PRIMARY KEY; -- This automatically: -- 1. Creates a new SERIAL column -- 2. Backfills existing rows with sequential values -- 3. Sets up auto-increment for new rows ``` After adding the primary key: ```bash postgres=# select * from no_primary_keys; key | value | pk -----+-------+---- bar | f | 2 foo | f | 1 (2 rows) ``` > 💡 **Pro Tip**: Your application code doesn't need any changes! You can continue inserting data without specifying the primary key: ```sql INSERT INTO no_primary_keys (key, value) VALUES ('qux', false); ``` ```bash postgres=# select * from no_primary_keys; key | value | pk -----+-------+---- bar | f | 2 foo | f | 1 qux | f | 3 (3 rows) ``` ## Need Help? 🤝 Have questions about adding primary keys to your tables? Reach out to us at [hi@artie.com](mailto:hi@artie.com)! # Data Processing Addendum Source: https://artie.com/docs/legal/data-processing-addendum Last updated: January 6, 2025 This Data Processing Addendum (“**DPA**”) amends and forms part of the written agreement between Customer and Artie Technologies Inc. (“**Artie**”) (collectively, “**the parties**”) for the provision of services to Customer (the “**Agreement**”). This DPA prevails over any conflicting term of the Agreement but does not otherwise modify the Agreement. 1. **Definitions** 1.1. In this DPA: “**Data Protection Law**” means all applicable laws, rules, regulations, and governmental requirements relating to the privacy, confidentiality, or security of Personal Data, as they may be amended or otherwise updated from time to time. “**Data Subject**” means a natural person whose Personal Data is Processed. “**Deidentified Data**” means data created using Personal Data that cannot reasonably be linked to such Personal Data, directly or indirectly. “**EEA**” means the European Economic Area. “**GDPR**” means Regulation (EU) 2016/679 (the “**EU GDPR**” or, where applicable, the “**UK GDPR**” as defined in section 3 of the DPA 2018. “**Personal Data**” means any information that relates, directly or indirectly, to an identified or identifiable natural person that Artie may Process on Customer’s behalf in performing the services under the Agreement. “**Processing**” (including its cognate "**Process**”) means any operation or set of operations which is performed on Personal Data or on sets of Personal Data, whether or not by automated means, such as collection, recording, organization, structuring, storage, adaptation or alteration, retrieval, consultation, use, disclosure by transmission, dissemination or otherwise making available, alignment or combination, restriction, erasure or destruction. “**Security Incident**” means a breach of security leading to the unauthorized or unlawful access by a third party, or confirmed accidental or unlawful destruction, loss or alteration, of Personal Data. “**Standard Contractual Clauses**” means (i) Module 2 of the Standard Contractual Clauses for the transfer of Personal Data to third countries pursuant to Regulation (EU) 2016/679 of the European Parliament and the Council approved by European Commission Implementing Decision (EU) 2021/914 of 4 June 2021, as currently set out at [https://eurlex.europa.eu/eli/dec\_impl/2021/914/oj](https://eurlex.europa.eu/eli/dec_impl/2021/914/oj) (the “**EU SCCs**”), and (ii) where the UK GDPR applies, the EU SCCs as supplemented by the International Data Transfer Addendum to the EU Commission Standard Contractual Clauses issued by the Commissioner under S119A(1) Data Protection Act 2018 (the “**UK SCCs**”). "**Swiss Data Protection Laws**" means the Swiss Federal Act Data Protection of 19 June 1992 and the Swiss Ordinance to the Swiss Federal Act on Data Protection of 14 June 1993, and any new or revised version of these laws that may enter into force for time to time. 1.2 Capitalized terms used but not defined herein have the meaning given to them in the Agreement. 2. **Scope and Roles** 2.1 The subject matter, nature and purpose of the Processing, the types of Personal Data and categories of Data Subjects are set out in **Annex I**. 2.2 Artie agrees that it will Process Personal Data only in accordance with the Agreement and this DPA. To the extent applicable, Artie will Process Personal Data as a “processor” or “service provider” and Customer shall act as a “controller” or “business”, as such terms are defined under applicable Data Protection Law. 3. **Data Protection** 3.1 When Artie Processes Personal Data, it will: (a) Process the Personal Data in accordance with Customer's documented instructions as described in the Agreement or this DPA. The Agreement and this DPA will generally constitute instructions for the Processing of Personal Data. Customer may issue further written instructions in accordance with this DPA. Artie will notify Customer if it considers that an instruction from Customer is in breach of Data Protection Law, unless it is prohibited from doing so by law; (b) provide reasonable assistance to Customer, taking into account the nature of the Processing and the information available to Artie, in complying with Customer's obligations to respond to requests concerning Personal Data from Data Subjects under applicable Data Protection Law; (c) implement and maintain appropriate physical, technical and organizational measures to ensure a level of security appropriate to the risk, which include the technical and organizational measures required by applicable Data Protection Law; (d) only entrust the Processing of Personal Data to personnel who have undertaken to comply with confidentiality requirements; and (e) upon termination of the Agreement, to the extent that Artie retains Personal Data, delete any copies of such Personal Data, provided that Artie shall permit Customer to obtain any copies of such Personal Data consistent with the functionality of the Services for a period of \[30] days after termination of the Agreement. 3.2 Artie certifies that it will not (a) “sell” (as defined in Data Protection Law) the Personal Data; (b) retain, use, or disclose the Personal Data for any purpose other than for the business purposes set out in this DPA and the Agreement; (c) retain, use, or disclose the Personal Data other than in the context of the direct relationship with Customer in accordance with the Agreement (d) except as otherwise permitted by applicable Data Protection Law, combine Personal Data provided by the Customer or otherwise disclosed in connection with the Agreement and this DPA with Personal Data that Artie receives from or on behalf of another person or persons, or collects from its own interaction with the Data Subject; or (e) share Personal Data with any third party for cross-context behavioral advertising. 4. **Customer Responsibilities** 4.1 Customer is responsible for the lawfulness of Personal Data processing under or in connection with the services. Customer will (i) provide all required notices and obtain all required consents, permissions and rights necessary under applicable Data Protection Law for Artie to lawfully Process Personal Data for the purposes contemplated by the Agreement; (ii) make appropriate use of the services to ensure a level of security appropriate to the particular content of the Personal Data; (iii) comply with all Data Protection Law applicable to the collection of Personal Data and the transfer of such Personal Data to Artie; and (iv) ensure its processing instructions comply with applicable laws (including applicable Data Protection Law). 5. **Subprocessing** 5.1 Customer agrees that Artie may use the third-party suppliers listed on its website at [https://artie.com/docs/legal/subprocessors](/legal/subprocessors) to Process Personal Data on its behalf for the provision of the services under the Agreement (each a “**Subprocessor**”). 5.2 Artie will maintain a list of Subprocessors and will provide at least 10 days’ prior notice of the addition or removal of any subprocessor, which may be given by posting detail of such addition or removal at the following URL: [https://artie.com/docs/legal/subprocessors](/legal/subprocessors). If Customer objects to the appointment of such Subprocessor within ten (10) days, it may terminate the portion of the services that cannot be provided without such Subprocessor on written notice to Artie that includes Customer’s legitimate and documented grounds for non-approval. 5.3 Artie will ensure that any Subprocessors to which it transfers Personal Data enter into written agreements with Artie requiring that the Subprocessor abide by terms substantially similar to those contained in this DPA. 5.4 Artie will remain liable for each Subprocessor’s compliance with the obligations under this DPA. 6. **Restricted Data Transfers** 6.1 The Standard Contractual Clauses shall apply to, and be incorporated by reference into this DPA in respect of, any transfers of Personal Data from the Customer (as “data exporter”) to Artie (as “data importer”) to the extent that: (a) the Customer’s Processing of Personal Data is subject to the GDPR or Swiss Data Protection Laws when making that transfer; or (b) the transfer is an “onward transfer” referred to in the Standard Contractual Clauses. 6.2 The Parties agree that execution of the Agreement shall have the same effect as signing the SCCs. 6.3 The Standard Contractual Clauses are further completed as follows: the optional docking clause in Clause 7 is implemented; Clause 9(a) option 2 is implemented and the time period therein is specified at Clause 5.2 of this DPA; the optional redress clause in Clause 11(a) is struck; the governing law in Clause 17 is the law of the Republic of Ireland; the court in Clause 18(b) are the Courts of the Republic of Ireland; and Annex 1 shall refer to Annex 1 of this DPA, Annex 1 (c) shall refer to the Irish Data Protection Commissioner, Annex 2 shall refer to Annex 2 of this DPA. 6.4 The UK SCCs shall form part of this DPA and apply to the transfer of Personal Data from the Customer (as data exporter) to Artie (as data importer) to the extent that: (a) the Customer’s Processing of Personal Data is subject to the UK GDPR when making that transfer; or (b) the transfer is an "onward transfer" as defined in the UK SCCs. 6.5 For the purposes of the UK SCCs: (a) the Addendum EU SCCs shall refer to the Standard Contractual Clauses as incorporated into this DPA; (b) Table 1 of the UK SCCs shall be completed with the details in paragraph A of Annex I; (c) the "Appendix Information" shall refer to the information set out in Annex I and Annex II; (d) for the purpose of Part 1, Table 4, the party that may end the UK SCCs in accordance with Section 19 of the UK Addendum is the data importer. 6.6 Annex III shall apply to the extent that Swiss Data Protection Laws apply to the Customer's Processing of Personal Data. 7. **Assistance and Notifications** 7.1 Upon Customer’s request, Artie will provide Customer with reasonable cooperation and assistance to the extent required to fulfill Customer’s obligation under applicable Data Protection Law, including to: (a) reply to investigations and inquiries from data protection regulators; and (b) carry out a data protection impact assessment related to the services, where Client does not otherwise have access to the relevant information necessary to perform such assessment. 7.2 Unless prohibited by applicable law, Artie must inform Customer without undue delay if Artie: (a) receives a request, complaint or other inquiry regarding the Processing of Personal Data; (b) receives a binding or non-binding request to disclose Personal Data from law enforcement, courts or any government body; (c) is subject to a legal obligation that requires Artie to Process Personal Data in contravention of Customer’s instructions; or (d) can no longer meet its obligations under Data Protection Law or this DPA. 7.3 Upon becoming aware of a Security Incident, Artie will inform Customer without undue delay and will provide timely information relating to the Security Incident as it becomes known or as is reasonably requested by Customer to allow Customer to fulfill its data breach reporting obligations under applicable Data Protection Law. 8. **Audit** 8.1 Artie will make available to Customer at Customer’s reasonable request information which is necessary to demonstrate compliance with this DPA and allow for any audits, including inspections, conducted by Customer or another auditor, as requested by Customer, provided that (a) such audits shall be carried out no more than once per year (unless otherwise required under applicable law) (b) on reasonable prior written notice and (c) during Artie’s normal business hours. 8.2 Customer may take reasonable and appropriate steps to: (a) ensure that Artie uses Personal Data in a manner consistent with Customer's obligations under Data Protection Law; and (b) upon reasonable notice, stop and remediate unauthorized use of Personal Data. 9. **Deidentified Data** 9.1 If Artie receives Deidentified Data from or on behalf of Customer, then Artie will: (a) take reasonable measures to ensure the information cannot be associated with a Data Subject. (b) publicly commit to Process the Deidentified Data solely in deidentified form and not to attempt to reidentify the information. (c) contractually obligate any recipients of the Deidentified Data to comply with the foregoing requirements and applicable Data Protection Law. 10. **General** 10.1 If there is any conflict between this DPA and the Agreement, this DPA will prevail to the extent of that conflict in connection with the Processing of Personal Data. 10.2 If any provision of this DPA is found by any court or administrative body of competent jurisdiction to be invalid or unenforceable, then the invalidity or unenforceability of such provision does not affect any other provision of this DPA and all provisions not affected by such invalidity or unenforceability will remain in full force and effect. 10.3 The parties hereby certify that they understand the requirements in this DPA and will comply with them. 10.4 The parties agree to negotiate in good faith any amendments to this DPA as may be required in connection with changes in application Data Protection Laws. 11. **Limitation of Liability** Other than any liability arising out of a breach of the Standard Contractual Clauses, each party’s liability, taken together in the aggregate, arising out of or related to this DPA, whether in contract, tort or under any other theory of liability, is subject to the ‘Limitation of Liability’ section of the Agreement, and any reference in such section to the liability of a party means the aggregate liability of that party under the Agreement, including this DPA. \\ ## ANNEX I 1. **LIST OF PARTIES** Customer is the controller and the data exporter and Artie is the processor and the data importer, in each case in relation to Processing of Personal Data in connection with the provision and receipt (as applicable) of the Services. 2. **DESCRIPTION OF TRANSFER** | **Subject Matter** | Artie’s provision of the services to Customer | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Duration of the Processing** | Personal Data will be retained only transiently or for a short duration to transmit the Personal Data from Customer’s chosen source to Customer’s chosen destination. Artie will process Customer Personal Data for the purposes of providing the services to Customer under the agreement. | | **Frequency of the Processing** | As and when the services are used. Duration of account/agreement life-cycle. | | **Categories of Data** | Any Personal Data selected by Customer in connection with Customer’s use of the services. The types of Personal Data processed are determined by Customer and may include without limitation: Name, Email address, Physical address, IP-address and other online identifiers, Date of birth, Telephone/mobile number, Location Data. | | **Special Categories of Data Processed** | The services are not intended to Process special categories of data. | | **Data Subjects** | Any data subjects of the Personal Data selected by Customer. | ## ANNEX II Artie shall implement and maintain the controls listed in this Annex II in accordance with industry standards generally accepted by information security professionals as necessary to reasonably protect Personal Data during storage, processing and transmission. **Physical access control** Technical and organizational measures to prevent unauthorized persons from gaining access to the data Processing systems available in premises and facilities (including databases, application servers and related hardware), where Personal Data are Processed, include: (a) establishing security areas, restriction of access paths; (b) establishing access authorizations for employees and third parties; (c) access control system (ID reader, magnetic card, chip card); (d) key management, card-keys procedures; (e) door locking (electric door openers etc.); (f) security staff, janitors; (g) surveillance facilities, video/CCTV monitor, alarm system; and (h) Securing decentralized data Processing equipment and personal computers. **Virtual access control** Technical and organizational measures to prevent data Processing systems from being used by unauthorized persons include: (a) user identification and authentication procedures; (b) ID/password security procedures (special characters, minimum length, change of password); (c) automatic blocking (e.g. password or timeout); (d) monitoring of break-in-attempts and automatic turn-off of the user ID upon several erroneous passwords attempts; (e) creation of one master record per user, user-master data procedures per data Processing environment; and (f) encryption of archived data media. **Data access control** Technical and organizational measures to ensure that persons entitled to use a data Processing system gain access only to such Personal Data in accordance with their access rights, and that Personal Data cannot be read, copied, modified or deleted without authorization, include: (a) internal policies and procedures; (b) control authorization schemes; (c) differentiated access rights (profiles, roles, transactions and objects); (d) monitoring and logging of accesses; (e) disciplinary action against employees who access Personal Data without authorization; (f) reports of access; (g) access procedure; (h) change procedure; (i) deletion procedure; and (j) encryption. **Disclosure control** Technical and organizational measures to ensure that Personal Data cannot be read, copied, modified or deleted without authorization during electronic transmission, transport or storage on storage media (manual or electronic), and that it can be verified to which companies or other legal entities Personal Data are disclosed, include: (a) encryption/tunneling; (b) logging; and (c) transport security. **Entry control** Technical and organizational measures to monitor whether Personal Data have been entered, changed or removed (deleted), and by whom, from data Processing systems, include: (a) logging and reporting systems; and (b) audit trails and documentation. **Control of instructions** Technical and organizational measures to ensure that Personal Data are Processed solely in accordance with the instructions of the Controller include: (a) unambiguous wording of the contract; (b) formal commissioning (request form); and (c) criteria for selecting the Processor. **Availability control** Technical and organizational measures to ensure that Personal Data are protected against accidental destruction or loss (physical/logical) include: (a) backup procedures; (b) mirroring of hard disks (e.g. RAID technology); (c) uninterruptible power supply (UPS); (d) remote storage; (e) antivirus/firewall systems; and (f) disaster recovery plan. **Separation control** Technical and organizational measures to ensure that Personal Data collected for different purposes can be Processed separately include: (a) separation of databases; (b) “internal Customer” concept / limitation of use; (c) segregation of functions (production/testing); and (d) procedures for storage, amendment, deletion, transmission of data for different purposes. ## ANNEX III This Annex III applies as set out in Clause 6.6 of this DPA. 1. **Interpretation** 1.1 Where this Addendum uses terms that are defined in the Standard Contractual Clauses, those terms will have the same meaning as in the Standard Contractual Clauses. In addition, the following terms have the following meanings: "**Addendum**" means this Annex III; "**Clauses**" means the Standard Contractual Clauses as incorporated into this DPA in accordance with Clause 6.1 and as further specified in Clause 6.3; and "**FDPIC**" means the Federal Data Protection and Information Commissioner. 1.2 This Addendum shall be read and interpreted in a manner that is consistent with Swiss Data Protection Laws, and so that it fulfils the Parties' obligation to provide appropriate safeguards as required by Article 46 GDPR and/or Article 6(2)(a) of the Swiss Data Protection Laws, as the case may be. 1.3 This Addendum will not be interpreted in a way that conflicts with rights and obligations provided for in Swiss Data Protection Laws. 1.4 Any references to legislation (or specific provisions of legislation) means that legislation (or specific provision) as it may change over time. This includes where that legislation (or specific provision) has been consolidated, re-enacted and/or replaced after this Swiss Addendum has been entered into. 1.5 In relation to any Processing of Personal Data subject to Swiss Data Protection Laws or to both Swiss Data Protection Laws and the GDPR, this Addendum amends and supplements the Clauses to the extent necessary so they operate: (a) for transfers made by the data exporter to the data importer, to the extent that Swiss Data Protection Laws apply to the data exporter’s Processing when making that transfer; and (b) to provide appropriate safeguards for the transfers in accordance with Article 6(2)(a) of the Swiss Data Protection Laws, as the case may be. 2. **Hierarchy** In the event of a conflict or inconsistency between this Addendum and the provisions of the Clauses or other related agreements between the Parties, existing at the time this Addendum is agreed or entered into thereafter, the provisions which provide the most protection to Data Subjects will prevail. 3. **Changes to the Clauses for transfers exclusively subject to Swiss Data Protection Laws** To the extent that the data exporter's Processing of Personal Data is exclusively subject to Swiss Data Protection Laws, or the transfer of Personal Data from a data exporter to a data importer under the Clauses is an "onward transfer" (as defined in the Clauses, as amended by the remainder of this paragraph 3.3(a)) the following amendments are made to the Clauses: (a) References to the "Clauses" or the "SCCs" mean this Swiss Addendum as it amends the SCCs. (b) Clause 6 Description of the transfer(s) is replaced with: "The details of the transfer(s), and in particular the categories of Personal Data that are transferred and the purpose(s) for which they are transferred, are those specified in Schedule 1 of this DPA where Swiss Data Protection Laws apply to the data exporter’s Processing when making that transfer." (c) References to "Regulation (EU) 2016/679" or "that Regulation" or ""GDPR" are replaced by "Swiss Data Protection Laws" and references to specific Article(s) of "Regulation (EU) 2016/679" or "GDPR" are replaced with the equivalent Article or Section of Swiss Data Protection Laws extent applicable. (d) References to Regulation (EU) 2018/1725 are removed. (e) References to the "European Union", "Union", "EU" and "EU Member State" are all replaced with "Switzerland". (f) Clause 13(a) and Part C of Annex I are not used; the "competent supervisory authority" is the FDPIC; (g) Clause 17 is replaced to state "These Clauses are governed by the laws of Switzerland". (h) Clause 18 is replaced to state: "Any dispute arising from these Clauses relating to Swiss Data Protection Laws will be resolved by the courts of Switzerland. A Data Subject may also bring legal proceedings against the data exporter and/or data importer before the courts of Switzerland in which he/she has his/her habitual residence. The Parties agree to submit themselves to the jurisdiction of such courts." (i) Until the entry into force of the revised Swiss Data Protection Laws, the Clauses will also protect Personal Data of legal entities and legal entities will receive the same protection under the Clauses as natural persons. 4. **Supplementary provisions for transfers of Personal data subject to both the GDPR and Swiss Data Protection Laws** 4.1 To the extent that the data exporter's Processing of Personal Data is subject to both Swiss Data Protection Laws and the GDPR, or the transfer of Personal Data from a data exporter to a data importer under the Clauses is an "onward transfer" under both the Clauses and the Clauses as amended by paragraph 3.3(c) of this Addendum: (a) for the purposes of Clause 13(a) and Part C of Annex I: (i) the FDPIC shall act as competent supervisory authority with respect to any transfers of Personal Data to the extent Swiss Data Protection Laws apply to the data exporter's Processing when making that transfer, or such transfer is an "onward transfer" as defined in the Clauses (as amended by paragraph 3.3 of this Addendum; and (ii) subject to the provisions of paragraph 2 of this Schedule 3 (UK Addendum), the supervisory authority identified in Schedule 1 shall act as competent supervisory authority with respect to any transfers of Personal Data to the extent the GDPR applies to the data exporter's processing, or such transfer is an "onward transfer" as defined in the Clauses. (b) the terms "European Union", "Union", "EU", and "EU Member State" shall not be interpreted in a way that excludes the ability of Data Subjects in Switzerland bringing a claim in their place of habitual residence in accordance with Clause 18(c) of the Clauses; and (c) Until the entry into force of the revised Swiss Data Protection Laws, the Clauses will also protect Personal Data of legal entities and legal entities will receive the same protection under the Clauses as natural persons. \\ # ELv2 License Addendum Source: https://artie.com/docs/legal/elv2-license-addendum Transfer by Artie uses Elastic License 2.0 **Elastic License 2.0 (ELv2)** **\*\*Acceptance\*\*** By using the software, you agree to all of the terms and conditions below. **\*\*Copyright License\*\*** The licensor grants you a non-exclusive, royalty-free, worldwide, non-sublicensable, non-transferable license to use, copy, distribute, make available, and prepare derivative works of the software, in each case subject to the limitations and conditions below **\*\*Limitations\*\*** You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software. You may not move, change, disable, or circumvent the license key functionality in the software, and you may not remove or obscure any functionality in the software that is protected by the license key. You may not alter, remove, or obscure any licensing, copyright, or other notices of the licensor in the software. Any use of the licensor’s trademarks is subject to applicable law. **\*\*Patents\*\*** The licensor grants you a license, under any patent claims the licensor can license, or becomes able to license, to make, have made, use, sell, offer for sale, import and have imported the software, in each case subject to the limitations and conditions in this license. This license does not cover any patent claims that you cause to be infringed by modifications or additions to the software. If you or your company make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company. **\*\*Notices\*\*** You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms. If you modify the software, you must include in any modified copies of the software prominent notices stating that you have modified the software.\\ **\*\*No Other Rights\*\*** These terms do not imply any licenses other than those expressly granted in these terms. **\*\*Termination\*\*** If you use the software in violation of these terms, such use is not licensed, and your licenses will automatically terminate. If the licensor provides you with a notice of your violation, and you cease all violation of this license no later than 30 days after you receive that notice, your licenses will be reinstated retroactively. However, if you violate these terms after such reinstatement, any additional violation of these terms will cause your licenses to terminate automatically and permanently. **\*\*No Liability\*\*** As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim. **\*\*Definitions\*\*** The \*licensor\* is the entity offering these terms, and the \*software\* is the software the licensor makes available under these terms, including any portion of it. \*you\* refers to the individual or entity agreeing to these terms. \*your company\* is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization. \*control\* means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise. Control can be direct or indirect. \*your licenses\* are all the licenses granted to you for the software under these terms. \*use\* means anything you do with the software requiring one of your licenses. \*trademark\* means trademarks, service marks, and similar rights. \\ # Privacy Policy Source: https://artie.com/docs/legal/privacy-policy Last updated: November 11, 2024 This notice is to inform visitors on how Artie Technologies Inc. (“***Artie***”) collects, use, and process the personal information we collect about you in the course of business on Artie’s website; artie.com and within our product dashboard, app.artie.com (“***Websites***”).\ \ If you have any concerns or questions related to our usage of your personal data, please contact us at [privacy@artie.com](mailto:privacy@artie.com), this is also linked at the bottom of our privacy notice. **Information you provide** Certain parts of our Websites may ask you to provide personal data voluntarily. For example, we may ask for your email and name in order to register and login to our dashboard. Another example for where we may ask for your contact details is in areas around support, subscribing to our newsletter and demo requests. The personal data you are asked to provide and the reasons you are asked to provide will be made clear at the point where we ask you to provide your personal data. **Information we automatically collect** When you visit our Websites, we may collect certain information automatically from your device. Some of this data may be considered personal data within the EU; specifically information that we collect may include your IP address, device type, unique device identification numbers, browser-type, general geographic location. We may also collect information regarding how you have interacted with our Websites. By collecting this information, it enables us to better understand and serve the visitors and customers who come to our Websites. **Information we receive from third parties** We may employ third-party companies and individuals from time to time, examples may include (but not limited to): * To facilitate our Service; * To provide the Service on our behalf; * To perform Service-related services; or * To assist us in analyzing how our Service is used. * To help us generate additional leads We want to inform our Website users that these third parties have access to your Personal Information. The reason is to perform the tasks assigned to them on our behalf. **Cookies** A cookie is a small data file that is stored on your computer or mobile device when you visit a website. When you visit the website again, the cookie allows that site to recognize your browser. Cookies may store user preferences and other information. You can set your browser to refuse all cookies or to indicate when a cookie is being sent. However, some website features or services may not function properly without cookies. We use cookies to understand and save your preferences for upcoming visits and compile aggregate data about site traffic and site interaction so that we can offer better site experiences and tools in the future. Any service providers used to help understand website visitors are not permitted to use the information collected on our behalf except to help us conduct and improve our business. **Links to Other Sites** Our Websites may contain links to other sites. If you click on a third-party link, you will be directed to that site. Note that these external sites are not operated by us. Therefore, we strongly advise you to review the Privacy Notice of these websites. We have no control over, and assume no responsibility for the content, privacy policies, or practices of any third-party sites or services. **How does Artie keep my personal data secure?** Artie stores all the personal information on leading and robust cloud service providers like Google Cloud Platform and Amazon Web Service. Artie uses industry-standard security protocols (such as encryption at REST, “Least Privilege” principle) to protect personal data. In addition our Websites are served with SSL and server software is continuously updated with the latest security patches. * Data at rest are all encrypted with AES256 or hashed with SHA256. * Data in transit are all secured via TLS. * Secrets management is centralized and encrypted with AGE where appropriate. **Changes to This Privacy Notice** We may update our Privacy Notice from time to time. Thus, we advise you to review this page periodically for any changes. **How to Contact Us** If you have any questions or suggestions about our Privacy Notice, do not hesitate to contact us. We can be reached at [privacy@artie.com](mailto:privacy@artie.com). # Subprocessors Source: https://artie.com/docs/legal/subprocessors Last updated: Oct 14, 2024 Artie Technologies Inc. (“***Artie***”) uses certain subprocessors to assist it in providing the Artie Services as described in the Master Service Agreement (“***MSA***”). Defined terms used herein shall have the same meaning as defined in the MSA. A subprocessor is any business that customer data may pass through as a side effect of using Artie’s service. We use partners for business processes that are not core to our expertise but are critical to our customers having a quality experience. | **Name** | **Related Services** | | ------------------- | ---------------------------------------------- | | Amazon Web Services | Networking and MSK and other hosting services. | | Datadog | Monitoring and logging. | # Terms of Service Source: https://artie.com/docs/legal/terms-of-service Last updated: November 25, 2024 This Artie Terms of Service Agreement (“***Agreement***”) is entered into by and between Artie Technologies Inc. (“***Artie***” or “***We***”) and the entity or person placing an order for or accessing the Artie Services (“***Customer***” or “***you***”). This Agreement consists of the terms and conditions set forth below, any terms referenced herein and any Artie Order Forms, online sign-up, or subscription flow that references this Agreement (the “***Order Form***”). If you are accessing or using the Artie Services on behalf of your company, you represent that you are authorized to accept this Agreement on behalf of your company, and all references to “you” or “Customer” reference your company. Please note that Artie may modify the terms and conditions of this Agreement in accordance with Section 12 (Modification of this Agreement). THIS AGREEMENT CONTAINS A BINDING, INDIVIDUAL ARBITRATION AND CLASS-ACTION WAIVER PROVISION. IF YOU ACCEPT THESE TERMS, YOU AND ARTIE AGREE TO RESOLVE DISPUTES IN BINDING, INDIVIDUAL ARBITRATION AND EACH GIVE UP THE RIGHT TO GO TO COURT INDIVIDUALLY OR AS PART OF A CLASS ACTION (SEE SECTION 19, BELOW). BY INDICATING YOUR ACCEPTANCE OF THIS AGREEMENT OR ACCESSING OR USING THE ARTIE SERVICES, YOU ARE AGREEING TO BE BOUND BY ALL TERMS, CONDITIONS AND NOTICES CONTAINED OR REFERENCED IN THIS AGREEMENT. IF YOU DO NOT AGREE TO THIS AGREEMENT, PLEASE DO NOT USE THE ARTIE SERVICES. FOR CLARITY, EACH PARTY EXPRESSLY AGREES THAT THIS AGREEMENT IS LEGALLY BINDING UPON IT. 1. **Artie Services Overview**. The “***Artie Services***” is a technology platform that allows you to manage, monitor and automate your data integration pipelines through use of one or more Artie Connectors (SaaS or On-Prem) or OS Connectors along with related services. The “***Artie Connectors***” are those source connectors and destination connectors listed on our website located at artie.com (the “***Site***”), as well as any custom connectors we may agree to build for you, all of which are and shall be licensed to you pursuant to either the Elv2 License Addendum (attached if applicable), or the terms and conditions of this Agreement. In addition, you may be able to use third party connectors that are available from the open source community (“***OS Connectors***”), which may be subject to the terms of a third party open source license or other terms and conditions governing their use. You are always free to use the Artie OS Connectors on without use of additional Artie Services. Additional descriptions concerning the functionality of the Artie Services are available at the Site. a. **On-Prem Software License**. With respect to any Order Form that includes on premise-installed data integration software product (“***On-Prem Software***”), subject to the terms of this Agreement, Artie grants to Customer a limited, personal, non-exclusive, non-transferable (except as part of a permitted assignment of this Agreement), non-sublicensable license during the subscription term of such Order Form to install, integrate and use for its own internal business purposes such On-Prem Software on the permitted type and number of computer hardware systems, storage platforms and computer frameworks. 2. **General Eligibility Requirements**. a. The individual signing on behalf of the Customer represents and warrants that he/she has the authority to bind the Customer to the Agreement. By accepting this Agreement, Customer represents and warrants to us: (i) that the Customer, if an individual, is at least 18 years of age; (ii) that the Customer has not previously been suspended or removed from using any portion of the Artie Services; and (iii) that the Customer’s registration and use of Artie Services is, and will continue to be, in compliance with any and all applicable laws and regulations. Moreover, Customer may not access the Artie Services if Customer is located in a territory where U.S. businesses are prohibited from engaging in business (such as Cuba, Iran, North Korea, Sudan, Syria) or if the Customer has been designated a Specially Designated National, Denied Person, or Denied Entity by the U.S. government. b. In order to access and use the Artie Services, Customer will need to register and create an account (“***Account***”). Customer agrees to provide accurate, current and complete information about the Account, which includes all individual sub-accounts (each an “***Authorized User Account***”) established for individuals who have been authorized by the Customer to access the Account and assigned a unique username-password combination to access and use the Artie Services on behalf of Customer (each an “***Authorized User***”). For more information on how Artie handles registration information, such as name, location, e-mail address or other contact information, and billing information, relating to Customer and any Authorized Users (“***Registration Information***”), please visit the Artie [Privacy Policy](/legal/privacy-policy). Artie reserves the right to suspend or terminate the Account or any individual Authorized User Account at any time if any information provided during the registration process or thereafter is or becomes inaccurate, false or misleading. In case Artie suspends or terminates the Account or any individual User Account, Artie shall provide Customer with notice of such suspension or termination and shall reinstate the Account or individual Authorized User Account once Customer demonstrates that the cause for suspension or termination has been cured. Customer is responsible for maintaining the confidentiality of Customer’s passwords and Customer’s Account, including all user names and passwords information assigned to its Authorized Users with respect to each Authorized User Account, and agrees to notify Artie if any of the passwords is lost, stolen, or disclosed to an unauthorized third-party, or otherwise may have been compromised. Customer is responsible for all activities that occur under the Account, including the activities carried out by individual employees, contractors or others. 3. **Permission to Use the Service**. a. **Right to Access Artie Services**. Subject to your compliance with this Agreement, Artie grants to you a limited, non-exclusive, non-transferable, non-sublicensable right during the term of this Agreement to access and use the features and functionalities of the Artie Services as set forth in the applicable Order Form. Certain of our downloadable software made available on or through your use of the Artie Services or via third parties may be provided to you under separate licensing terms, rather than the terms and conditions of this Agreement. The applicable terms and conditions shall be disclosed to you where the downloadable software is made available to you within the Artie Services. As stated above, all Artie Connectors are licensed to you either under the ELv2 License Addendum (attached if applicable), or the terms and conditions of this Agreement, unless otherwise agreed upon in the Order Form. b. **Beta Releases and Features**. Artie may provide you with “alpha”, “beta”, or other early-stage components of the Artie Services, connectors, integrations, or features (“***Beta Releases***”). Artie makes no promises that future versions of Beta Releases will be released or will be made available under the same commercial terms. Artie may terminate your right to use any Beta Releases at any time in Artie’s sole discretion, without liability. In case Artie terminates your right to use any Beta Releases, Artie shall provide Customer with notice of such termination. WITH RESPECT TO BETA RELEASES, CUSTOMER ACKNOWLEDGES AND AGREES THAT BETA RELEASES MAY NOT BE COMPLETE OR FULLY FUNCTIONAL AND MAY CONTAIN BUGS, ERRORS, OMISSIONS, SECURITY VULNERABILITIES, AND OTHER PROBLEMS FOR WHICH ARTIE WILL NOT BE RESPONSIBLE. ACCORDINGLY, ANY USE OF BETA RELEASES ARE AT CUSTOMER’S SOLE RISK NOTWITHSTANDING ANYTHING TO THE CONTRARY HEREIN. c. **Use Outside of the United States**. Artie and the Artie Services are operated from the United States. Artie does not make any representations or warranties that the Artie Services will meet all of the legal and/or regulatory requirements of any non-U.S. jurisdiction and Artie expressly disclaims all liability in connection with the foregoing. d. **Modification of the Artie Services**. We reserve the right to modify or discontinue certain features of the Artie Services at any time. Artie shall provide notice to Customer of any such modifications. e. **Additional Terms**. Your use of certain features of Artie Services may be subject to additional terms and conditions that Artie will disclose to you through the Artie Services or that you otherwise acknowledge as part of your downloading of any applicable software or applications related to the Artie Services (the “***Additional Terms***”), such terms and conditions for any downloadable software or applications that Artie may offer and service level agreements concerning the Artie Services (“***SLAs***”). 4. **Fees and Payment**. a. **Calculation of Fees**. You agree to pay the fees set forth in the applicable Order Form. Our third-party payment services providers may charge additional fees for the use of their services, including fees in connection with cross-border monetary transactions. Before you are required to pay any fees, you will have an opportunity to review and accept the applicable fees that you will be charged, including the time-period in which those fees will be charged. All fees are in U.S. Dollars and are non-refundable, unless otherwise stated in this Agreement. Artie or its third-party payment service providers may change any fees, including by adding fees, on a going-forward basis at any time. Artie or its third-party payment service providers will charge the payment method you specify at the time of purchase (the “***Payment Method***”), which you represent and warrant that you are authorized to use. You authorize Artie to charge all sums described in your Order Form to that Payment Method. If you pay any applicable fees with a credit card, Artie may seek pre-authorization of your credit card account prior to your purchase to verify that the credit card is valid and has the necessary funds or credit available to cover your purchase. If your account does not have sufficient funds or your credit card declines a charge for the fees due, Artie may refuse your use and access of Artie Services. If Customer’s use of any Artie Services exceeds the usage or capacity set forth on the applicable Order Form, or otherwise requires the payment of additional Fees (per the terms of this Agreement), Artie will invoice Customer in arrears for such additional usage or capacity and Customer agrees to pay the additional Fees in the manner provided herein. b. **Payment Terms**. Artie will bill for the Artie Services through an invoice, through the marketplace where the original purchase was made, or directly through credit card if provided. Full payment for invoices must be received by Artie within 30 days after the issuance of the invoice (which may be sent by email). Unpaid amounts are subject to a finance charge of 1.5% per month on any outstanding balance, or the maximum permitted by law, whichever is lower, plus all expenses of collection and may result in immediate termination of Service. If Customer is paying by credit card, Customer represents and warrants that it has the right to use the credit card provided and grants Artie the right to provide the credit card information, including the credit card number, its expiration date and billing address, to third parties for the purposes of facilitating payment transactions. Artie reserves the right to charge a 3% surcharge for any credit card payments. Verification of information may be required prior to the acknowledgment or completion of any payment transaction. If Customer believes that Artie has billed Customer incorrectly, Customer must contact Artie no later than 60 days after the date of the first billing statement in which the error or problem appeared, in order to receive an adjustment or credit. Inquiries should be directed to Artie’s customer support department. c. **Taxes**. Fees do not include taxes, levies, duties or similar governmental assessments of any nature (“***Taxes***”). Customer is responsible for paying all Taxes associated with its purchases hereunder. If Artie has the legal obligation to pay or collect Taxes for which Customer is responsible under this Section, Artie will invoice Customer and Customer will pay that amount unless Customer provides Artie with a valid tax exemption certificate authorized by the appropriate taxing authority. Taxes will not be deducted from payments to Artie, except as required by applicable law, in which case Customer will increase the amount payable as necessary so that, after making all required deductions and withholdings, Artie receives and retains (free from any liability for Taxes) an amount equal to the amount it would have received had no such deductions or withholdings been made. Upon Artie’s request, Customer will provide to Artie its proof of withholding tax remittance to the respective tax authority. Where applicable, Customer will provide its VAT/GST Registration Number(s) on the Order Form to confirm the business use of the ordered services. d. **On-Premise Usage Calculation**. With respect to On-Prem Software, if Customer’s system configuration blocks Artie’s ability to determine the monthly amounts owed for Customer’s use of such On-Prem Software, then at Artie’s request Customer will provide a monthly report and such other reasonably requested information in order for Artie to determine such amounts, including a breakdown of Users, data sources and destinations between Customer-internal usage and powered by Artie usage. For On-Prem Software provided on a fixed monthly or annual fee, such monthly report shall not be applicable. 5. **Billing Subscription; Automatic Renewal**. If you are on a billing subscription plan, your subscription is subject to auto-renewal on the same terms and subscription length set forth in the Order Form unless and until you cancel your subscription or Artie terminates it. Your Payment Method will be charged automatically upon the next renewal of your subscription all applicable fees for the next subscription period. You must cancel your subscription before the end of each subscription period in order to avoid billing of the next periodic subscription fees to your Payment Method. Artie will bill the periodic subscription fee to the Payment Method you provide to Artie during registration (or to a different Payment Method if you change your account information). 6. **Free Trials; Discounts**. Artie may make available trials (for duration indicated from time to time by Artie) for access to the Artie Services without charge to you (each a “***Free Trial***”). Moreover, from time to time, Artie may offer you discounts or other credits based on campaigns Artie may run from time to time (e.g., customer referral credits, discounts on annual renewals, etc.) (each a “***Promotion***”). You will be required to create an Account for the Artie Services in order to register for a Free Trial or Promotion. If you wish to continue using the Artie Services at the end of the Free Trial, you will be required to enter a payment method which will be charged in accordance with Sections 4 and 5. If you cancel your Account, any rights to any Promotion shall automatically terminate. 7. **Prohibited Conduct**. By using the Site or the Artie Services, you agree not to: a. use the Artie Services for any illegal purpose, or in violation of any local, state, national, or international law; b. violate, or encourage others to violate, the rights of third parties, including by infringing or misappropriating third party intellectual property rights; c. copy, modify or distribute the Artie Services (unless otherwise expressly permitted by Artie); or remove any copyright, trademark or other proprietary rights notices contained in or on the Artie Services or any content made available on or through the Artie Services; d. interfere with security-related features of the Artie Services, including without limitation by (i) disabling or circumventing features that prevent or limit use or copying of any content, or (ii) reverse engineering or otherwise attempting to discover the source code of the Artie Services or any part thereof except to the extent that such activity is expressly permitted by applicable law; e. interfere with the operation of the Artie Services or any other user's use of the Artie Services, including without limitation by (i) uploading or otherwise disseminating viruses, adware, spyware, worms, or other malicious code, (ii) making unsolicited offers or advertisements to other users of the Artie Services, (iii) attempting to collect, personal information about users or third parties without their consent; or (iv) interfering with or disrupting any networks, equipment, or servers connected to or used to provide the Artie Services, or violating the regulations, policies, or procedures of those networks, equipment, or servers; f. perform any fraudulent activity including impersonating any person or entity, claiming false affiliations, accessing the Artie Services accounts of others without permission, or falsifying your age or date of birth; g. sell or otherwise transfer the access granted herein or any Materials (as defined in Section 14 below) or any right or ability to view, access, or use any Materials; or h. attempt to do any of the foregoing in this Section 7, or assist or permit any persons in engaging in any of the activities described in this Section 7. i. with respect to On-Prem Software, (i) translate or localize any On-Prem Software; (ii) redistribute, encumber, sell, rent, lease, sublicense, display, publish, disclose or otherwise transfer rights to any On-Prem Software, in whole or in part, to any third party; or (iii) allow any third party (other than contractors that are Users) to access or use the On-Prem Software. 8. **Linked Sites and Community-Developed Services**. The Artie Services may include links to other websites or services (“***Linked Sites***”) solely as a convenience to you and other users (e.g., links to websites where you may select OS Connectors, third-party websites where your data is stored, etc.). The Artie Services may provide tools that enable the importation and/or exportation of your information to Linked Sites. By using these tools, you agree that Artie may transfer such information to the applicable Linked Sites. Linked Sites are not under our control, Artie is not responsible for their use of your imported or exported information. Furthermore, the Artie Services may require you to download and/or install certain third party components in order to access and use certain features within the Artie Services, including OS Connectors and other open source, third-party or community-developed features, components, or services (collectively, “***Community-Developed Connectors/Services***”). Customer acknowledges that Artie does not review and is not responsible for the Community-Developed Connectors/Services, including without limitation the security of the Community-Developed Connectors/Services or the fitness of Community-Developed Connectors/Services for any purpose, and your use of any Community-Developed Connectors/Services or Linked Sites may be subject to additional third party terms and conditions. Artie makes no express or implied warranties with regard to the Community-Developed Connectors/Services, or any information, materials, products, or services that are contained on or accessible through Linked Sites. ACCESS AND USE OF COMMUNITY-DEVELOPED CONNECTORS/SERVICES AND LINKED SITES, INCLUDING THE INFORMATION, MATERIAL, PRODUCTS, AND SERVICES ON COMMUNITY-DEVELOPED CONNECTORS/SERVICES OR LINKED SITES OR AVAILABLE THROUGH COMMUNITY-DEVELOPED CONNECTORS/SERVICES OR LINKED SITES, IS SOLELY AT YOUR OWN RISK. 9. **Termination**. a. Either party may terminate this Agreement in the event the other party materially breaches the Agreement and, to the extent that such breach can be cured, fails to cure such breach within thirty (30) days after receipt of notice by the other party. b. We may terminate this Agreement or suspend or terminate your Artie Account at any time for any reason. We will notify you within thirty (30) days of termination of these Service by email or at the next time you attempt to access your Artie Account. c. If Artie terminates this Agreement, due to a breach by you of this Agreement, any prepaid fees shall be wholly non-refundable. If you terminate this Agreement due to a material breach by Artie, you shall receive a pro-rata refund of any prepaid fees for Artie Services not yet received. d. You may terminate your Account in accordance with the applicable Order Form by providing written notice within thirty (30) days to customer service at [hi@artie.com](mailto:hi@artie.com). If you terminate your Account, you will remain obligated to pay all outstanding fees, if any, relating to your use of the Artie Services incurred prior to termination and any prepaid fees shall be wholly non-refundable except as expressly set forth herein. e. Upon any termination or expiration of this Agreement, whether by you or Artie, any information that you have submitted or uploaded on or through the service or that which is related to your Account may no longer be accessed by you and Artie will have no obligation to maintain any information in its databases or to forward any information to you or any third party. Upon any termination of this Agreement, any provision that by its nature or express terms should survive will survive such termination or expiration. 10. **Feedback; OS Contributions**. If you provide any comments, suggestions or other feedback to Artie regarding the Artie Services (“Feedback”), you acknowledge that the Feedback is not confidential and you authorize Artie to use that Feedback without restriction and without payment to you. Accordingly, you hereby grant to Artie a nonexclusive, royalty-free, fully-paid, perpetual, irrevocable, transferable, and fully sublicensable right to use and otherwise freely exercise and exploit the Feedback in any manner and for any purpose. Any modifications, changes, and enhancements you make to any of the Artie Connectors shall be licensed to Artie pursuant to the terms of the ELv2 license, if applicable. 11. **Customer Data**. a. Protections for Customer Data; Personal Data. Excluding Registration Information, Customer owns any data and information it integrates via data pipelines connected through the Artie Connectors (“***Customer Data***”). Customer hereby grants to Artie a non-exclusive, worldwide, royalty-free, fully paid up, sublicensable (through multiple tiers), right and license during the term of the Agreement to use the Customer Data strictly to perform Artie’s obligations under this Agreement, including to provide the Artie Services and to ensure the Artie Services are working as intended. Artie treats any Customer Data as Customer confidential information. Artie will implement commercially reasonable efforts at all times to protect Customer Data from unauthorized use, access, or disclosure in the same manner that Artie uses to protect its own confidential information of a similar nature. You are solely responsible for obtaining all rights, consents and permissions necessary for Artie to process Customer Data in connection with the Artie Services. To the extent that Customer Data includes any information that reasonably relates, directly or indirectly, to an identified or identifiable natural person (“***Personal Data***”), the parties agree to comply with their respective obligations under the Artie [Data Processing Addendum](/legal/data-processing-addendum), which is hereby incorporated into this Agreement. b. Special Data. In the event that Customer intends to integrate any Customer Data considered to be sensitive or otherwise subject to specific protections under applicable laws exceeding any requirements that apply to Personal Data generally, such as, for illustrative purposes, information that is regulated by the Health Information Portability and Accountability Act (“***HIPAA***”), the Payment Card Industry Data Security Standard (“***PCI DSS***”), the Gramm-Leach-Bliley Act (“***GLBA***”) and other U.S. federal, state, or foreign laws applying specific security standards (collectively, “***Special Data***”), Customer shall evaluate whether the technical and organizational measures described in the Data Processing Addendum are sufficient to protect Special Data. Customer shall not integrate any Special Data to the Artie Services if Customer determines that such technical and organizational measures are insufficient to protect Special Data in accordance with applicable laws. c. Customer Responsibilities. Customer is responsible for security relating to its environment, particularly its computer hardware systems, storage platforms and computer frameworks, and security relating to its configuration of the Artie Services. This includes implementing and managing procedural, technical, and administrative safeguards on its systems and networks sufficient to: (a) ensure the confidentiality, security, integrity, and privacy of Customer Data and (b) follow the principle of least privilege when connecting the Artie Services to Customer’s computer hardware systems, storage platforms and computer frameworks, especially by granting no more than read-only access to data sources. Customer is also responsible for provisioning Users, including: (i) methods of authenticating Users (such as SSO or industry-standard secure username/password policies); (ii) managing admin privileges; (iii) deauthorizing personnel who no longer need access to the applicable Artie Services; and (iv) setting up any API usage in a secure way. Artie will have no obligations or liability as to any loss resulting from Customer’s security configuration or administration of the Artie Services. d. Usage Data. Customer agrees that Artie has the right to aggregate, collect and analyze data and information relating to the Artie Services (“***Usage Data***”) and shall be free (during and after the term hereof) to (i) use Usage Data and other information to improve Artie’s products and services, and (ii) disclose Usage Data and other information solely in an aggregated and de-identified format that does not identify Customer or any individual. 12. **Modification of this Agreement**. Artie may update the terms and conditions of this Agreement (which may include changes pricing and plans) from time to time with prior notice to Customer in accordance with notice provisions in Section 19. The updated version of this Agreement will be available at [https://artie.com/docs/legal/terms-of-service](/legal/terms-of-service). Following such notice, your continued use of the Artie Services on or after the date the updated version of this Agreement is effective and binding, as indicated at the top of this Agreement, constitutes your acceptance of the updated version of this Agreement. If you do not agree to the updated version of this Agreement, this Agreement will be terminated immediately, and if you are using a paid version of the Artie Services, you may elect to receive a pro-rata refund of any prepaid fees applicable to Artie Services not yet received. The updated version of this Agreement supersedes all prior versions. No waiver will be implied from conduct or failure to enforce or exercise rights under this Agreement. Waivers must be made in writing and executed by an authorized representative of the waiving party. 13. **Ownership; Proprietary Rights**. The Artie Service is owned and operated by Artie. The visual interfaces, graphics, design, compilation, information, data, computer code (including source code or object code), products, software, services, and all other elements of the Artie Services (the “***Materials***”) provided by Artie (through use and access of the Artie Services or otherwise) are protected by all relevant intellectual property and proprietary rights and applicable laws. All Materials contained in Artie Services are the property of Artie or the corresponding third-party licensors (for clarification, all Artie Connectors are owned by Artie and licensed in accordance with the ELv2 license unless otherwise specified in an Order Form). Artie reserves all rights to the Materials (other than the OS Connectors or other Materials licensed to you pursuant to separate third party terms and conditions) not granted expressly in this Agreement. 14. **Indemnity**. a. **Indemnity by Artie**. Artie agrees to defend, indemnify, and hold harmless you and your officers, directors, employees, consultants, affiliates, subsidiaries and agents from and against all third party claims, liabilities, damages, losses, and expenses, including reasonable attorneys' fees and costs, arising out of or in any way connected with infringement by the Artie Services of any U.S. patent, copyright, trademark or trade secret. In response to any claim or potential claim of infringement, if required by settlement or injunction, or if Artie determines these actions are reasonably necessary to avoid material liability, Artie may at its option: (i) procure a license for the affected portion of the Artie Services; (ii) modify the Artie Services so as to avoid infringement but be materially equivalent; or (iii) terminate the Order Form for the affected Artie Services and refund any fees you have pre-paid for Artie Services not yet received. Notwithstanding the above, Artie’s obligations under this Section do not apply to the extent infringement results from: (a) modification of the Artie Service by someone other than Artie or its subcontractors; (b) combination of the Artie Service with other third party services where the Artie Service would not by itself be infringing (c) use of any version of the Artie Service other than the current release; (d) any OS Connectors; (e) in the case of On-Prem Software, that are modified by or on behalf of Customer after delivery by Artie; (f) in the case of On-Prem Software, where Customer continues the allegedly infringing activity after being notified thereof or after being provided an update that would have avoided the alleged infringement, or (g) where Customer’s use of the Artie Services is not strictly in accordance with this Agreement. In addition, Artie’s obligations under this Section shall not apply to the extent you are using a Free Trial or Beta Releases. THIS SECTION STATES YOUR SOLE AND EXCLUSIVE REMEDY OF AND THE ENTIRE LIABILITY OF ARTIE, OR ANY OF ITS SUPPLIERS, OFFICERS, DIRECTORS, EMPLOYEES, SHAREHOLDERS, CONTRACTORS OR REPRESENTATIVES, WITH RESPECT TO ANY CLAIM OF INTELLECTUAL PROPERTY INFRINGEMENT. b. **Indemnity by Customer**. You agree that you will be responsible for your use of the Artie Services, and you agree to defend, indemnify, and hold harmless Artie and its officers, directors, employees, consultants, affiliates, subsidiaries and agents (collectively, the “***Artie Entities***”) from and against any and all claims, liabilities, damages, losses, and expenses, including reasonable attorneys' fees and costs, arising out of or in any way connected with (i) your access to, use of, or alleged use of the Artie Services; (ii) your violation of this Agreement or any representation, warranty, or agreements referenced herein, or any applicable law or regulation; (iii) your violation of any third-party right, including without limitation any intellectual property right, publicity, confidentiality, property or privacy right; or (iv) any disputes or issues between you and any third party. c. **Indemnification Procedures**. Each party’s defense and indemnification obligations are subject to the indemnifying party receiving: (i) prompt written notice of a claim; (ii) the exclusive right to control and direct the investigation, defense and settlement of the claim; and (iii) all reasonable necessary cooperation of the indemnified party at the indemnifying party’s expense (as to reasonable out-of- pocket costs). The indemnifying party must not settle any claim without the indemnified party’s prior written consent if the settlement would require the indemnified party to admit fault, pay amounts that the indemnifying party must pay under this Section, or take or refrain from taking any action. The indemnified party may participate in a claim through counsel of its own choosing at its own expense.\ \\ 15. **Warranty Disclaimers**. a. TO THE EXTENT PERMITTED BY LAW, THE ARTIE SERVICES, ANY OTHER MATERIAL, AND ALL CONTENT RELATED THERETO ARE PROVIDED “AS IS” AND “AS AVAILABLE,” AND ARTIE MAKES NO WARRANTIES OR CONDITIONS, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABILITY, MERCHANTABLE QUALITY, INFRINGEMENT, TITLE, QUIET ENJOYMENT, AND FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTIES ARISING FROM A COURSE OF DEALING OR USAGE IN TRADE, TOGETHER WITH SIMILAR WARRANTIES, WHETHER ARISING UNDER ANY LAW OR OTHERWISE. SPECIFICALLY AND WITHOUT LIMITING THE FOREGOING IN ANY WAY, ARTIE SPECIFICALLY DISCLAIMS ANY WARRANTY OR CONDITION: (i) THAT ANY SERVICE OR PRODUCT OFFERED BY ARTIE WILL MEET ANY PARTICULAR REQUIREMENTS, INCLUDING THAT SUCH SERVICES AND PRODUCTS SHALL BE FREE OF SECURITY VULNERABILITIES; (ii) THAT SERVICES OR PRODUCTS OFFERED BY ARTIE WILL BE UNINTERRUPTED, TIMELY, SECURE, ERROR-FREE, OR THAT ANY DEFECTS IN ANY PRODUCT WILL BE CORRECTED; OR (iii) RELATING TO THE ACCURACY OR RELIABILITY OF THE RESULTS OBTAINED THROUGH YOUR USE OF THE ARTIE SERVICES, ANY MATERIALS OR ANY COMMUNICATIONS, DATA, INFORMATION, OR CONTENT DOWNLOADED OR OTHERWISE OBTAINED OR ACQUIRED THROUGH THE USE OF ANY OF THE FOREGOING. THE PARTIES AGREE, AND IT IS THEIR INTENTION, THAT IN NO EVENT SHALL ANY WARRANTY PROVIDED BY LAW APPLY UNLESS REQUIRED TO APPLY BY APPLICABLE STATUTE NOTWITHSTANDING THEIR EXCLUSION BY CONTRACT. TO THE EXTENT THAT ARTIE CANNOT DISCLAIM ANY SUCH WARRANTY AS A MATTER OF APPLICABLE LAW, THE SCOPE AND DURATION OF SUCH WARRANTY SHALL BE LIMITED TO THE FULLEST EXTENT PERMITTED BY LAW. b. CUSTOMER IS SOLELY RESPONSIBLE FOR TESTING ANY ON-PREM SOFTWARE BEFORE USING IT IN A LIVE ENVIRONMENT, THE PROTECTION OF ITS DATA AND INFORMATION THROUGH INSTALLATION OF THE MOST RECENT COMPUTER VIRUS DETECTION PROGRAMS, AND THE TIMELY CREATION OF BACK-UP COPIES. NOTWITHSTANDING ANYTHING TO THE CONTRARY IN THIS AGREEMENT, ARTIE DOES NOT WARRANT AND EXPRESSLY DISCLAIMS THAT ANY ON-PREM SOFTWARE LICENSED WILL BE SECURE OR AVAILABLE AT ANY PARTICULAR TIME OR LOCATION, AND THE RESULTS OF USING THE ON-PREM SOFTWARE WILL MEET CUSTOMER’S REQUIREMENTS. ARTIE SHALL NOT BE LIABLE FOR ANY DAMAGE (INCLUDING FOR LOSS OR CORRUPTION OF DATA), OR RESPONSIBLE FOR RESTORATION OF SUCH DATA, IN EACH CASE TO THE EXTENT SUCH DAMAGE RESULTS FROM CUSTOMER’S INSUFFICIENT TESTING OF ANY ON-PREM SOFTWARE PRIOR TO DEPLOYMENT IN A LIVE ENVIRONMENT. c. SOME JURISDICTIONS DO NOT PERMIT THE DISCLAIMER OF CERTAIN IMPLIED WARRANTIES, SO CERTAIN OF THE FOREGOING DISCLAIMERS MAY NOT APPLY TO YOU. TO THE EXTENT THAT ARTIE CANNOT DISCLAIM ANY SUCH WARRANTY AS A MATTER OF APPLICABLE LAW, THE SCOPE AND DURATION OF SUCH WARRANTY WILL BE THE MINIMUM PERMITTED UNDER SUCH LAW. 16. **Limitation of Liability**. a. IN NO EVENT WILL ARTIE BE LIABLE TO YOU FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, GOODWILL, USE, DATA, OR OTHER INTANGIBLE LOSSES) ARISING OUT OF OR RELATING TO YOUR ACCESS TO OR USE OF, OR YOUR INABILITY TO ACCESS OR USE, THE ARTIE SERVICES OR ANY MATERIALS OR CONTENT WITHIN THE ARTIE SERVICES, WHETHER BASED ON WARRANTY, CONTRACT, TORT (INCLUDING NEGLIGENCE), STATUTE OR ANY OTHER LEGAL THEORY, WHETHER OR NOT THE ARTIE ENTITIES HAVE BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGE. b. YOU AGREE THAT THE AGGREGATE LIABILITY OF THE ARTIE ENTITIES TO YOU FOR ANY AND ALL CLAIMS ARISING OUT OF RELATING TO THE USE OF OR ANY INABILITY TO USE THE ARTIE SERVICES (INCLUDING ANY MATERIALS OR CONTENT AVAILABLE THROUGH THE SERVICE) OR OTHERWISE UNDER THIS AGREEMENT, WHETHER IN CONTRACT, TORT, OR OTHERWISE, IS LIMITED TO THE GREATER OF (I) THE AMOUNTS YOU HAVE PAID TO ARTIE FOR ACCESS TO AND USE OF THE ARTIE SERVICES IN THE 6 MONTHS PRIOR TO THE CLAIM OR (II) \$50. c. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES. ACCORDINGLY, THE ABOVE LIMITATION MAY NOT APPLY TO YOU. EACH PROVISION OF THIS AGREEMENT THAT PROVIDES FOR A LIMITATION OF LIABILITY, DISCLAIMER OF WARRANTIES, OR EXCLUSION OF DAMAGES IS TO ALLOCATE THE RISKS UNDER THIS AGREEMENT BETWEEN THE PARTIES. THIS ALLOCATION IS AN ESSENTIAL ELEMENT OF THE BASIS OF THE BARGAIN BETWEEN THE PARTIES. EACH OF THESE PROVISIONS IS SEVERABLE AND INDEPENDENT OF ALL OTHER PROVISIONS OF THIS AGREEMENT. THE LIMITATIONS IN THIS SECTION 17 WILL APPLY EVEN IF ANY LIMITED REMEDY FAILS OF ITS ESSENTIAL PURPOSE. 17. **Governing Law; Forum**. This Agreement shall be governed by the laws of the State of California without regard to conflict of law principles. To the extent that any lawsuit or court proceeding is permitted hereunder (and not deemed subject to arbitration pursuant to Section 19), you and Artie agree to submit to the personal and exclusive jurisdiction of the state courts and federal courts located within San Francisco, California for the purpose of litigating all disputes. 18. **Dispute Resolution US and Canadian Residents**. If there is a dispute, our user support team is happy to try to resolve the issue. If that does not work and your principal place of business is in the United States or Canada, your options in the case of a dispute arising out of or related to this Agreement are to (a) go to small claims court (if applicable) or (b) bring a claim in binding arbitration as described in this Section 18; you may not bring that claim in another court or participate in a non-individual class action claim against Artie. This Dispute Resolution section applies only if Customer that have a principal place of business in the United States or Canada or, if the Customer is an individual, if you live in the United States of Canada. For all other Customers, Section 20 hereof shall apply in lieu of this Section 18. a. Small Claims. Either of the parties can bring a claim in small claims court in (a) San Francisco, California, (b) the county where you live, or (c) another place the parties both agree on, as long as it qualifies to be brought in that court. b. Arbitration. If any dispute cannot be resolved amicably and informally and a small claims action is not initiated, Artie and you agree to resolve any claims related to these Terms (or our other legal terms) through final and binding arbitration, regardless of the type of claim or legal theory. If one of the parties brings a claim in court that should be arbitrated and the other party refuses to arbitrate it, the other party can ask a court to force both parties to go to arbitration (compel arbitration). Either party can also ask a court to halt a court proceeding while an arbitration proceeding is ongoing. With respect to any arbitration proceeding the following process shall be followed: i. Any disputes involving a claim or claims of ten thousand dollars (\$10,000) or less, in the aggregate must be resolved through binding, non-appearance-based arbitration (i.e., arbitration conducted online, through written filings, and/or via teleconference) (each a “***Sub 10K Arbitration***”). ii. Any Sub 10K Arbitration shall be conducted through the American Arbitration Association (the “***AAA***”) in accordance with its Commercial Arbitration Rules, Consumer Due Process Protocol, and Supplementary Procedures for Resolution of Consumer-Related Disputes. The AAA rules are available at [www.adr.org](http://www.adr.org) or by calling 1-800-778-7879. iii. The following rules will apply to any Sub 10K Arbitration: (a) the arbitration will be conducted by telephone, online, or based solely on written submissions (at the choice of the party seeking relief); (b) the arbitration must not involve any personal appearance by the parties or witnesses (unless Artie and you agree otherwise); and (c) any judgment on the arbitrator’s rendered award may be entered in any court with competent jurisdiction. iv. Disputes that involve a claim (or claims) of more than ten thousand dollars (\$10,000) in the aggregate must be resolved per the AAA’s rules about whether the arbitration hearing has to be in-person. To the extent permitted by the AAA’s applicable rules, upon your request Artie will agree to conduct the arbitration by telephone or online. v. Any judgment rendered by the arbitrator in any arbitration proceeding may be entered in any court having jurisdiction and the arbitrator's decision shall be final and legally binding. c. No Class Actions. Artie and you each agree that each party can only bring claims against the other on an individual basis. This means: (a) neither party can bring a claim as a plaintiff or class member in a class action, consolidated action, or representative action; (b) an arbitrator cannot combine multiple people’s claims into a single case (or preside over any consolidated, class, or representative action); and (c) an arbitrator’s decision or award in one person’s case can only impact that user, not other users, and can’t be used to decide other users’ disputes. If a court decides that this “No class actions” clause isn’t enforceable or valid, then this entire Dispute Resolution for US and Canadian Residents section will be null and void, but the rest of the Terms will still apply and the Dispute Resolution for non-US and non-Canadian Residents of these Terms shall apply instead to you with respect to dispute resolution as if you were not a resident of the United States or Canada. d. Changes to this Section. Notwithstanding the terms of Section 18 of this Agreement, if Artie changes this Dispute Resolution for US and Canadian Residents section after the date you entered into this Agreement or accepted an updated version of this Agreement pursuant to Section 18 (whichever is later), you may reject any such change by providing Artie written notice of such rejection by mail or hand delivery to: Artie Technologies Inc. Attn: 601 Van Ness Ave Apt 28, San Francisco, CA 94102, or by email from the email address associated with your account to: [hi@artie.com](mailto:hi@artie.com), within 30 days of the date such change became effective in accordance with Section 18. To be effective, the notice must include your full name (or the full name of the entity on whose behalf you are submitted the notice) and clearly indicate your intent to reject changes to this Dispute Resolution for US and Canadian Residents section. By rejecting changes, you are agreeing that you will arbitrate any dispute between you and Artie in accordance with the provisions of this Dispute Resolution for US and Canadian Residents section as of the date entered into this Agreement or or accepted an updated version of this Dispute Resolution for US and Canadian Residents section pursuant to Section 18 and this Section 18 (whichever is later). 19. **General**. a. Assignment. Neither party may assign or transfer this Agreement, in whole or in part, by operation of law or otherwise, without the other party’s prior written consent; provided that each party may assign this Agreement in connection with a merger, acquisition, or sale of all or substantially all of its business or assets related to this Agreement. b. Severability. If any provision of this Agreement is held invalid or unenforceable by a court of competent jurisdiction, the remaining provisions of this Agreement will remain in full force and effect, and the provision affected will be construed so as to be enforceable to the maximum extent permissible by law. c. Notices. All notices required or permitted under this Agreement will be in writing, will reference this Agreement, and will be deemed given: (i) when delivered personally; (ii) one (1) business day after deposit with a nationally-recognized express courier, with written confirmation of receipt; (iii) three (3) business days after having been sent by registered or certified mail, return receipt requested, postage prepaid; or (iv) in the case of notices to You that are sent via electronic mail to the contact person indicated in your Order Form, twenty-four (24) hours after sending. All such notices will be sent to the addresses set forth above or to such other addresses as may be specified by either party to the other party in accordance with this Section. d. Waiver. The failure to require performance of any provision will not affect our right to require performance at any time thereafter, nor shall a waiver of any breach or default of this Agreement or any provision of this Agreement constitute a waiver of any subsequent breach or default or a waiver of the provision itself. e. Entire Agreement. This Agreement, together with all Order Forms the Privacy Policy, any applicable Additional Terms, any applicable SLA, and any other agreements expressly incorporated by reference herein, constitute the entire and exclusive understanding and agreement between you and Artie regarding your use of and access to the Artie Services, and except as expressly permitted above may be amended only by a written agreement signed by authorized representatives of all parties to this Agreement. f. Section Headings. Use of section headers in this Agreement is for convenience only and shall not have any impact on the interpretation of particular provisions. g. Force Majeure. Neither party will be responsible for any failure or delay in its performance under this Agreement due to causes beyond its reasonable control, including, but not limited to, labor disputes, strikes, lockouts, shortages of or inability to obtain labor, energy, raw materials or supplies, war, acts of terror, riot, acts of God or governmental action. # Analytics Portal Source: https://artie.com/docs/monitoring/analytics-portal We also have a [Datadog integration](https://docs.datadoghq.com/integrations/artie/) so you can see the same metrics and build your own dashboards and monitors in Datadog. Once you have a running deployment, you will be able to see metrics in the [Analytics Portal](https://app.artie.com/analytics). ## Available filters For each of these metrics, you also have the ability to set a filter: 1. Deployment (All or specific deployment) 2. Specific tables (This is available if you specify the deployment) 3. Whether to include or exclude backfills 4. Mode (Replication or history) ## Metrics The following metrics are available in the Analytics Portal: * Rows synced over time * Type of operation (create, update, delete, backfill) * Table distribution * Ingestion lag based on time * Ingestion lag based on number of rows * For Postgres, we'll also graph your replication slot size * For Snowflake, we'll monitor your virtual warehouse's queued and execution time ![Example of what Analytics Portal looks like](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/analytics_portal.png) # Available Metrics Source: https://artie.com/docs/monitoring/available-metrics In this section, we will go over the metrics that Artie Transfer emits and the future roadmap. ## Today Artie Transfer's currently only supports metrics and integrates with Datadog. We are committed to being vendor neutral, but not at the cost of stability and reliability. As such, we will be using [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-go#project-status) when the library is stable. We also plan to support application tracing such that we can directly plug into your APM provider. ## Metrics You can specify additional tags in the configuration file and have it apply to all the metrics that Transfer emits. Click [here](/open-source/running-artie/options#telemetry) for more details. | Name | Description | Unit | Tags | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | ----------------------------------------------------------------- | | `*.row.lag` | Difference between Kafka's high watermark (which is at the partition level) and the current message offset. | - | `groupid`, `topic`, `partition`, `table`, `mode` | | `*.ingestion.lag.95percentile` | p95 of time lag from Kafka message was published and received.

Since Transfer `1.4.6` | `ms` | `groupid`, `topic`, `partition`, `table`, `mode` | | `*.ingestion.lag.avg` | Avg of time lag from Kafka message was published and received.

Since Transfer `1.4.6`

If self-hosting Transfer, this is a good metric to monitor. | `ms` | `groupid`, `topic`, `partition`, `table`, `mode` | | `*.ingestion.lag.max` | max lag from Kafka message was published and received.

Since Transfer `1.4.6` | `ms` | `groupid`, `topic`, `partition`, `table`, `mode` | | `*.process.message.count` | How many rows has Transfer processed. | Count | `database`, `schema`, `table`, `groupid`, `op`, `skipped`, `mode` | | `*.process.message.95percentile` | p95 of how long each row process takes. | `ms` | `database`, `schema`, `table`, `groupid`, `op`, `skipped`, `mode` | | `*.process.message.avg` | Average of how long each row process takes. | `ms` | `database`, `schema`, `table`, `groupid`, `op`, `skipped`, `mode` | | `*.process.message.max` | Max of how long each row process takes. | `ms` | `database`, `schema`, `table`, `groupid`, `op`, `skipped`, `mode` | | `*.process.message.median` | Median of how long each row process takes. | `ms` | `database`, `schema`, `table`, `groupid`, `skipped`, `mode` | | `*.flush.count` | How many flush operations have been performed. | Count | `database`, `schema`, `table`, `what`, `reason` | | `*.flush.95percentile` | p95 of how long each flush process takes. | `ms` | `database`, `schema`, `table`, `what`, `reason` | | `*.flush.avg` | Avg of how long each flush process takes. | `ms` | `database`, `schema`, `table`, `what`, `reason` | | `*.flush.max` | Max of how long each flush process takes. | `ms` | `database`, `schema`, `table`, `what`, `reason` | | `*.flush.median` | Median of how long each flush process takes. | `ms` | `database`, `schema`, `table`, `what`, `reason` | ## The what tag explained The `what` tag aims to provide a high level of visibility into whether an attempt has succeeded or not. And if it did not succeed, it will provide additional visibility into which particular operation failed (vs just providing a generic error state). Transfer will provide `what:success` if the attempt failed and different reasoning depending on the error state. This way, our monitors and response to failures can be more actionable and we can jump straight to the offending code block. ![Visualization of the what tag](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/telemetry_what_tag.png) # Schema Changes Source: https://artie.com/docs/monitoring/schema-changes Stay up to date on schema changes in your source database. ## Notification settings By default, every user has schema change notifications enabled. You can toggle this setting by going to [Settings > My settings](https://app.artie.com/settings?tab=user) and altering the `Receive table change notifications` setting. ## Schema changes Our schema change job runs every weekday at 6 AM PST. We will notify you if there's any changes in your source database. We will skip the notification if there aren't any. By default, you'll receive this via email, and you can [configure Slack notifications](/account/enabling-slack-notifications) as well. Changes we monitor: 1. Tables added 2. Tables removed 3. Tables changed * Columns added * Columns removed * Columns altered (data type, default value and not null constraint) ![Example of what schema changes looks like](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/schema_change_example.png) # Examples Source: https://artie.com/docs/open-source/running-artie/examples ## Introduction Below, you can see sample configuration files to describe different workloads. To see all the available settings, please see the [options](/open-source/running-artie/options) page. ## Postgres options ### Postgres -> Snowflake ```yaml outputSource: snowflake kafka: bootstrapServer: kafka:9092 groupID: group_abc topicConfigs: - db: shop tableName: customers schema: inventory topic: "dbserver1.inventory.customers" cdcFormat: debezium.postgres.wal2json cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter snowflake: account: ACCOUNT_ID username: USER password: PASSWORD warehouse: DWH_NAME region: us-east-2.aws ``` ### Postgres -> BigQuery ```yaml outputSource: bigquery kafka: bootstrapServer: kafka:9092 groupID: group_abc topicConfigs: - db: shop tableName: customers schema: inventory topic: "dbserver1.inventory.customers" cdcFormat: debezium.postgres.wal2json cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter bigquery: pathToCredentials: PATH_TO_CREDENTIALS projectID: PROJECT_ID defaultDataset: DEFAULT_DATASET ``` ## MongoDB options ### MongoDB -> Snowflake ```yaml outputSource: snowflake kafka: bootstrapServer: kafka:9092 groupID: group_abc topicConfigs: - db: shop tableName: customers schema: inventory topic: "dbserver1.inventory.customers" cdcFormat: debezium.mongodb cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter snowflake: account: ACCOUNT_ID username: USER password: PASSWORD warehouse: DWH_NAME region: us-east-2.aws ``` ### MongoDB -> BigQuery ```yaml outputSource: bigquery kafka: bootstrapServer: kafka:9092 groupID: group_abc topicConfigs: - db: shop tableName: customers schema: inventory topic: "dbserver1.inventory.customers" cdcFormat: debezium.mongodb cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter bigquery: pathToCredentials: PATH_TO_CREDENTIALS projectID: PROJECT_ID defaultDataset: DEFAULT_DATASET ``` ## Optional Blocks ### Error Reporting to Sentry If you provide your Sentry DSN; Artie Transfer will automatically report any errors during processing into your Sentry project. Visit this [link](https://docs.sentry.io/product/sentry-basics/dsn-explainer/) to see how you can find your Sentry DSN. ```yaml reporting: sentry: dsn: https://docs.sentry.io/product/sentry-basics/dsn-explainer/ ``` Transfer is using a vendor neutral logging library, so file a [feature request](https://github.com/artie-labs/transfer/issues/new) if you use Rollbar, or another vendor! ### Telemetry Click [here](/monitoring/available-metrics) to see all the metrics that Transfer emits. ```yaml telemetry: metrics: provider: datadog settings: tags: - env:production - customer:artie.so namespace: "transfer." addr: "127.0.0.1:8125" ``` # Options Source: https://artie.com/docs/open-source/running-artie/options This page describes the available configuration settings for Artie Transfer. ```bash /transfer -c /path/to/config.yaml ``` *Note: Keys here are formatted in dot notation for readability purposes, please ensure that the proper nesting is done when writing this into your configuration file. To see sample configuration files, visit the* [examples](/examples) page. | Key | Optional | Description | | ---------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mode` | Y | Defaults to `replication`. Supported values are currently: | | `outputSource` | N | This is the destination.
Supported values are currently: | | `queue` | Y | Defaults to `kafka`.
Other valid options are `kafka` and `pubsub`.

Please check the respective sections below on what else is required. | | `reporting.sentry.dsn` | Y | DSN for Sentry alerts. If blank, will just go to stdout. | | `flushIntervalSeconds` | Y | Defaults to `10`.

Valid range is between `5 seconds` to `6 hours`. | | `bufferRows` | Y | Defaults to `15,000` | | `flushSizeKb` | Y | Defaults to `25mb`
| ## Source ### Kafka | Key | Optional | Description | | ----------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kafka.bootstrapServer` | N | Pass in the Kafka bootstrap server. For best practices, pass in a comma separated list of bootstrap servers to maintain high availability. This is the [same spec as Kafka](https://kafka.apache.org/documentation/#producerconfigs_bootstrap.servers). | | `kafka.groupID` | N | This is the name of the Kafka consumer group. You can set to whatever you'd like. Just remember that the offsets are associated to a particular consumer group. | | `kafka.username` | Y | If you'd like to use SASL/SCRAM auth, you can pass the username and password. | | `kafka.password` | Y | If you'd like to use SASL/SCRAM auth, you can pass the username and password. | | `kafka.enableAWSMSKIAM` | Y | Enable this if you would like to use IAM authentication to communicate with Amazon MSK. If enabled, please ensure `AWS_REGION`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set. | | `kafka.disableTLS` | Y | Enable this to disable TLS. | | `kafka.topicConfigs` | N | Follow the same convention as `*.topicConfigs` below. | ```yaml kafka: bootstrapServer: localhost:9092,localhost:9093 groupID: transfer username: artie password: transfer enableAWSMSKIAM: false ``` ### Google Pub/Sub If you don't have access to Kafka, you can use Google Pub/Sub. However, we would recommend you to use Kafka for the best possible experience. | Key | Optional | Description | | | -------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | | `pubsub.projectID` | N | This is your GCP Project ID, click [here](../destinations/bigquery.md#getting-your-project-identifier) to see how you can find it. | | | `pubsub.pathToCredentials` | N | This is the path to the credentials for the service account to use. You can re-use the same credentials as BigQuery, or you can use a different service account to support use cases of cross-account transfers. | | | `pubsub.topicConfigs` | N | Follow the same convention as `*.topicConfigs` below. | | ```yaml pubsub: projectID: 123 pathToCredentials: /path/to/pubsub.json topicConfigs: - {} ``` ### Topic Configs `topicConfigs` are used at the table level and store configurations like: * Destination's database, schema and table name. * What does the data format look like? * Whether it should do row based soft deletion or not. * Whether it should drop deleted columns or not. | Key | Optional | Description | | --------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `*.topicConfigs[0].db` | N | Name of the database in destination. | | `*.topicConfigs[0].tableName` | Y | Name of the table in destination. | | `*.topicConfigs[0].schema` | N | Name of the schema in Snowflake.

Not needed for BigQuery. | | `*.topicConfigs[0].topic` | N | Name of the Kafka topic. | | `*.topicConfigs[0].cdcFormat` | N | Name of the CDC connector (thus format) we should be expecting to parse against.

The supported values are:
  1. `debezium.postgres`
  2. `debezium.mongodb`
  3. `debezium.mysql`
| | `*.topicConfigs[0].cdcKeyFormat` | N | Format for what Kafka Connect will the key to be.
This is called `key.converter` in the Kafka Connect properties file.

The supported values are:
If not provided, the default value will be `org.apache.kafka.connect.storage.StringConverter`. | | `*.topicConfigs[0].dropDeletedColumns` | Y | Defaults to `false`.

When set to `true`, Transfer will drop columns in the destination when Transfer detects that the source has dropped these columns. This column should be turned on if your organization follows standard practice around database migrations. | | `*.topicConfigs[0].softDelete` | Y | Defaults to `false`.

When set to `true`, Transfer will add an additional column called `__artie_delete` and will set the column to true instead of issuing a hard deletion. | | `*.topicConfigs[0].skippedOperations` | Y | Comma-separated string for Transfer to specified operations.

Valid values are: Can be specified like: `c,d` to skip create and deletes. | | `*.topicConfigs[0].includeArtieUpdatedAt` | Y | Defaults to `false`.

When set to `true`, Transfer will emit an additional timestamp column named `__artie_updated_at` which signifies when this row was processed. | | `*.topicConfig[0].includeDatabaseUpdatedAt` | Y | Defaults to `false`.

When set to `true`, Transfer will emit an additional timestamp column called `__artie_db_updated_at` which signifies the database time of when the row was processed. | | `*.topicConfigs[0].bigQueryPartitionSettings` | Y | Enable this to turn on BigQuery table partitioning. | ```yaml bigQueryPartitionSettings: partitionType: time partitionField: ts partitionBy: daily ``` | Key | Optional | Description | | ------------------------------------------ | -------- | -------------------------------------------------------------------------------------------------------------------- | | `bigQueryPartitionSettings.partitionType` | N | Type of partitioning. We currently support only time-based partitioning. The valid values right now are just `time`. | | `bigQueryPartitionSettings.partitionField` | N | Which field or column is being partitioned on. | | `bigQueryPartitionSettings.partitionBy` | N | This is used for time partitioning, what is the time granularity? Valid values right now are just `daily` | ## Destination ### BigQuery | Key | Optional | Description | | ---------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bigquery.pathToCredentials` | N | Path to the credentials file for Google.

You can also directly inject `GOOGLE_APPLICATION_CREDENTIALS` ENV VAR, else Transfer will set it for you based on this value provided. | | `bigquery.projectID` | N | Google Cloud Project ID | | `bigquery.location` | Y | Location of the BigQuery dataset.

Defaults to `us`. | | `bigquery.defaultDataset` | N | The default dataset used.

This just allows us to connect to BigQuery using data source notation (DSN). | ### Databricks | Key | Optional | Description | | --------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | `databricks.host` | N | Host URL e.g. `https://test-cluster.azuredatabricks.net` | | `databricks.httpPath` | N | HTTP path of the SQL warehouse | | `databricks.port` | Y | HTTP port of the SQL warehouse (defaults to 443) | | `databricks.catalog` | N | Unity catalog name | | `personalAccessToken` | N | Personal access token for Databricks | | `volume` | N | Volume name for Databricks. Volume must exist under the database and schema that you are replicating into. | ### Microsoft SQL Server | Key | Optional | Description | | ---------------- | -------- | ------------------------------------------------------------------ | | `mssql.host` | N | Database host e.g. `test-cluster.us-east-1.redshift.amazonaws.com` | | `mssql.port` | N | - | | `mssql.database` | N | Name of the database | | `mssql.username` | N | - | | `mssql.password` | N | - | ### S3 | Key | Type | Optional | Description | | ----------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `s3.bucket` | String | N | S3 bucket name. Example: `artie-transfer`. | | `s3.folderName` | String | Y | Optional folder name within the bucket. If this is specified, Artie Transfer will save the files under `s3://artie-transfer/folderName/...` | | `s3.awsAccessKeyID` | String | N | The `AWS_ACCESS_KEY_ID` for the service account. | | `s3.awsSecretAccessKey` | String | N | The `AWS_SECRET_ACCESS_KEY` for the service account. | ```yaml s3: bucket: artie-transfer folderName: foo # Files will be saved under s3://artie-transfer/foo/... awsAccessKeyID: AWS_ACCESS_KEY_ID awsSecretAccessKey: AWS_SECRET_ACCESS_KEY ``` ### Snowflake | Key | Optional | Description | | --------------------- | -------- | ---------------------- | | `snowflake.account` | N | Account Identifier | | `snowflake.username` | N | Snowflake username | | `snowflake.password` | N | Snowflake password | | `snowflake.warehouse` | N | Virtual warehouse name | | `snowflake.region` | N | Snowflake region | ### Redshift | Key | Optional | Description | | ---------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `redshift.host` | N | Host URL e.g. `test-cluster.us-east-1.redshift.amazonaws.com` | | `redshift.port` | N | - | | `redshift.database` | N | Namespace / Database in Redshift. | | `redshift.username` | N | - | | `redshift.password` | N | - | | `redshift.bucket` | N | Bucket for where staging files will be stored.
[Click here](/open-source/guides/running-redshift#setting-up-s3-bucket) to see how to set up a S3 bucket and have it automatically purged based on expiration. | | `redshift.optionalS3Prefix` | Y | The prefix for S3, say bucket is `foo` and prefix is `bar`. It becomes: s3://foo/bar/file.txt | | `redshift.credentialsClause` | N | Redshift [credentials clause](https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-authorization.html) to store staging files into S3. | ## Telemetry Overview of Telemetry can be found [here](/monitoring/available-metrics). | Key | Type | Optional | Description | | ------------------------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `telemetry.metrics` | Object | Y | Parent object. See below. | | `telemetry.metrics.provider` | String | Y | Provider to export metrics to. Transfer currently only supports: datadog. | | `telemetry.metrics.settings` | Object | Y | Additional settings block, see below | | `telemetry.metrics.settings.tags` | Array | Y | Tags that will appear for every metric like: env:production, company:foo | | `telemetry.metrics.settings.addr` | String | Y | Address for where the statsD agent is running. Defaults to 127.0.0.1:8125 if none is provided. | | `telemetry.metrics.settings.sampling` | Number | Y | Percentage of data to send. Provide a number between 0 and 1. Defaults to 1 if none is provided. Refer to this for additional information. | # Overview Source: https://artie.com/docs/open-source/running-artie/overview In this section, we will go over how to install and run Artie Transfer. ## Architecture ```mermaid graph LR S((Source)):::db --> A[Artie Reader] A[Artie Reader] --> B[Kafka] B[Kafka] --> C[Artie Transfer] C[Artie Transfer] --> D((Destination)):::db classDef db fill:#ffffde,stroke:#333,stroke-width:2px; ``` ## Install You can install Reader and Transfer through: 1. [Docker](https://hub.docker.com/r/artielabs/) 2. GitHub - [Transfer](https://github.com/artie-labs/transfer) and [Reader](https://github.com/artie-labs/reader) ## How do I specify the configuration file? To run Reader or Transfer, you can do so by specifying the configuration file by using the `--config` option. **Running this as a standalone binary** ```bash /transfer --config config.yaml ``` ## Next To see all the available settings, please click on the link below to continue. Various configuration options to run Artie # null Source: https://artie.com/docs/quickstart Welcome to Artie, the database replication platform. ![Elephant greeting](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/welcome.png) ### What is Artie? Artie is a database replication platform that helps you sync data from your source databases to your destinations in real-time with change data capture. ### Creating your first deployment ✨ 1. [Select your source database](#1-select-your-source-database) 2. [Select the tables you want to replicate](#2-select-the-tables-you-want-to-replicate) 3. [Select your destination](#3-select-your-destination) Click **+ New deployment** and specify the source you want Artie to replicate from. #### 1. Select your source database How would you like Artie to connect to your source? Artie's fixed IPs, instructions for setting up SSH tunnels, and more. Instructions along with service account scripts. #### 2. Select the tables you want to replicate Select the tables you'd like Artie to replicate. Additionally, you can enable history mode for specific tables that you would like to have additional tracking. Within each table, you can also click on the ⚙️ icon to specify advanced settings. Specify settings like table alias, column exclusion, merge predicates, and more. #### 3. Select your destination Choose your destination and provide destination-specific details. Once the destinations have been filled out, you can optionally change any of these advanced settings. When enabled, Artie will add a new column `__artie_updated_at` to your tables. This is useful for incremental models. When enabled, Artie will add a new column `__artie_db_updated_at` to your tables. This is the database time for when the transaction occurred. By default, Artie will not drop any columns. If this is enabled, we will start to drop deleted columns. By default, Artie will hard delete rows. If this is enabled, we will start to soft delete rows. Soft deleted rows will have `__artie_deleted` set to `true`. ### Monitoring Once your deployment is live, you can monitor its status from the analytics portal and stay up to date on your schema changes from our schema change notifications. **Analytics Portal** **Schema change notifications** # DocumentDB Source: https://artie.com/docs/sources/documentdb Learn how to use Artie to replicate data from DocumentDB via change streams. To run Artie with DocumentDB, you **must** have SSH tunnels enabled as DocumentDB only allows access within your VPC. See [SSH tunneling](/docs/connection-options) for further instructions. ## Required settings * Cluster endpoint * Port (default is `27017`) * Service account * Database name (ensure the database has change stream enabled) You can find the cluster endpoint by going to the AWS console, selecting the DocumentDB cluster and go to the "Configuration" tab. ![DocumentDB cluster endpoint](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/documentdb.png) ```mongodb // Creating a service account for Artie to subscribe to change stream db.createUser({user: "artie", pwd: "changeme", roles: ["read"]}); // Grants access to DocumentDB change streams use admin; db.grantRolesToUser("artie", [ { role: "readAnyDatabase", db: "admin" } ]); // Depending on the permission granularity you want, you can use the following commands to grant permissions to the service account. // Enable change streams for all collections in database "changeme" db.adminCommand({ modifyChangeStreams: 1, database: "changeme", collection: "", enable: true }); // Enable change streams for all collections in all databases db.adminCommand({ modifyChangeStreams: 1, database: "", collection: "", enable: true }); // Advanced: Enable change streams for the collection "foo" in database "changeme" db.adminCommand({ modifyChangeStreams: 1, database: "changeme", collection: "foo", enable: true }); ``` # DynamoDB Source: https://artie.com/docs/sources/dynamodb ## Required settings * DynamoDB Streams ARN (view type must be set to `NEW_IMAGE` or `NEW_AND_OLD_IMAGES`) * Service account ![DynamoDB Streams](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/dynamodb_streams.png) ```hcl provider "aws" { region = "us-east-1" } resource "aws_iam_role" "dynamodb_streams_role" { name = "DynamoDBStreamsRole" assume_role_policy = jsonencode({ Version = "2012-10-17", Statement = [ { Action = "sts:AssumeRole", Principal = { Service = "ec2.amazonaws.com" }, Effect = "Allow", Sid = "" } ] }) } resource "aws_iam_policy" "dynamodb_streams_access" { name = "DynamoDBStreamsAccess" description = "My policy that grants access to DynamoDB streams" policy = jsonencode({ Version = "2012-10-17", Statement = [ { Effect = "Allow", Action = [ "dynamodb:GetShardIterator", "dynamodb:DescribeStream", "dynamodb:GetRecords", "dynamodb:ListStreams", // Required for export "dynamodb:DescribeTable", "dynamodb:ListExports", "dynamodb:DescribeExport", "dynamodb:ExportTableToPointInTime" ], // Don't want to use "*"? You can specify like this: // Resource = [ TABLE_ARN, TABLE_ARN + "/stream/*" ] Resource = "*" # Modify this to restrict access to specific streams or resources }, // Export (snapshot) requires access to S3 { "Effect" : "Allow", "Action" : [ "s3:ListBucket" ], "Resource" : "arn:aws:s3:::artie-transfer-test" }, { "Effect" : "Allow", "Action" : [ "s3:GetObject", // Required for export "s3:PutObject", "s3:GetBucketLocation" ], "Resource" : "arn:aws:s3:::artie-transfer-test/*" } ] }) } resource "aws_iam_role_policy_attachment" "dynamodb_streams_role_policy_attachment" { role = aws_iam_role.dynamodb_streams_role.name policy_arn = aws_iam_policy.dynamodb_streams_access.arn } output "service_role_arn" { value = aws_iam_role.dynamodb_streams_role.arn } resource "aws_iam_user" "dynamodb_streams_user" { name = "dynamodb-artie-user" path = "/" } resource "aws_iam_user_policy_attachment" "user_dynamodb_streams_attachment" { user = aws_iam_user.dynamodb_streams_user.name policy_arn = aws_iam_policy.dynamodb_streams_access.arn } resource "aws_iam_access_key" "dynamodb_streams_user_key" { user = aws_iam_user.dynamodb_streams_user.name } output "aws_access_key_id" { value = aws_iam_access_key.dynamodb_streams_user_key.id sensitive = true } output "aws_secret_access_key" { value = aws_iam_access_key.dynamodb_streams_user_key.secret sensitive = true } ``` ## Backfills In order to backfill data from a DynamoDB table, we will perform a data export to S3 and then import the data into the target table. This is more favorable as it does not consume read capacity units (RCUs) from the source table, and you will only need to pay for data transfer costs (which are a fraction of RCUs). # Microsoft SQL Server Source: https://artie.com/docs/sources/microsoft-sql-server ## Required settings * Host name * Port (default is `1433`) * Service account * Database name * CDC enabled for database and tables to replicate ```sql USE DATABASE_NAME; CREATE LOGIN artie WITH PASSWORD = 'PASSWORD'; CREATE USER artie FOR LOGIN artie; ALTER LOGIN artie ENABLE; GRANT SELECT on DATABASE::DATABASE_NAME to artie; GO; ``` In order to enable CDC for SQL Server, you will enable it at the **database and table** level. 1. **Enable CDC for your database** ```sql USE DATABASE_NAME; EXEC sys.sp_cdc_enable_db; -- If you're using RDS: EXEC msdb.dbo.rds_cdc_enable_db DATABASE_NAME; -- Now specify the retention period of your CDC logs, retention is specified as mins. -- We recommend you setting this between 24h to 7 days EXEC sys.sp_cdc_change_job @job_type = N'cleanup', @retention = 10080; -- 7 days GO; ``` 2. **Enable CDC for your tables** ```sql -- You can specify the service_account as the @role_name to restrict access EXEC sys.sp_cdc_enable_table @source_schema = 'SCHEMA_NAME', @source_name = 'TABLE_NAME', @role_name = null; ``` ### Troubleshooting By default, `sys.sp_cdc_enable_table` will use the primary keys of the table as the unique identifiers. If you are running into issues with this, you can optionally set `@index_name` to an unique index of your choice. For example, if you had a table that looked like this ```sql CREATE TABLE orders ( id INTEGER IDENTITY(1001,1) NOT NULL PRIMARY KEY, order_date DATE NOT NULL, purchaser INTEGER NOT NULL, quantity INTEGER NOT NULL, product_id INTEGER NOT NULL, FOREIGN KEY (purchaser) REFERENCES customers(id), FOREIGN KEY (product_id) REFERENCES products(id) ); ``` If you don't want to use `id`, you can do something like this: ```sql ALTER TABLE orders ADD COLUMN prefix VARCHAR(255) DEFAULT 'orders' NOT NULL; -- Create unique index CREATE UNIQUE INDEX cdc_index ON dbo.orders (id, prefix); -- When enabling CDC, use cdc_index instead of the primary key EXEC sys.sp_cdc_enable_table @source_schema = 'dbo', @source_name = 'orders', @role_name = NULL, @index_name = 'cdc_index'; ``` # MongoDB Source: https://artie.com/docs/sources/mongodb Learn how to use Artie to replicate data from MongoDB via change streams. MongoDB server **must** be in a replica set. If your deployment only has a `standalone` server, you can create a replica set with one member. Need help? Check out this [guide](https://www.mongodb.com/docs/manual/tutorial/convert-standalone-to-replica-set/). ## Required settings * Connection string * Service account ### Connection string 1. Go to Atlas UI 2. Find your deployment and click "Connect" We support both MongoDB SRV format or standard connection string. ![MongoDB connection string](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/mongodb_connection_string.png) ### Service account You can create a service account through the Atlas UI or by running a script. * Click on `Database Access` on the left * Click on `Add New Database User` * Under `Database User Privileges`, open `Built-in Role` and Select `Only read any database` ![MongoDB Atlas](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/mongodb_atlas.png) ```js /* If the user does not exist. */ use admin; db.createUser({ user: "artie", pwd: "", roles: ["readAnyDatabase", { role: "read", db: "local" }] }); /* If the user already exists */ db.updateUser("artie", { roles: ["readAnyDatabase", { role: "read", db: "local" }] }); ``` ### Advanced If you are replicating a MongoDB collection into a partitioned table downstream, you will want to consider enabling this so that the full document before change is available for deletes. This is because we will need the previous row to grab the partitioned field(s) in order to select the right partition downstream. To enable this, you'll want to run the following commands: ```js // Enable preAndPostImage on the replica set use admin; db.runCommand({ setClusterParameter: { changeStreamOptions: { preAndPostImages: { expireAfterSeconds: 100 } } } }); // Enable preAndPostImage on the collection use databaseName; db.runCommand({ collMod:"collectionName",changeStreamPreAndPostImages: { enabled: true } }); // See the previous setting db.adminCommand( { getClusterParameter: "changeStreamOptions" } ); // To disable this behavior, you can set expiredAfterSeconds to off db.runCommand( { setClusterParameter: { changeStreamOptions: { preAndPostImages: { expireAfterSeconds: 'off' } } } } ); ``` # MySQL Source: https://artie.com/docs/sources/mysql ## Required settings * Host name * Port (default is `3306`) * Service account * Database name * Database with `binlog_format` set to `ROW` * Database with `binlog retention hours` set to at least 24 hours ```sql CREATE USER 'artie_transfer' IDENTIFIED BY 'password'; GRANT SELECT, RELOAD, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'artie_transfer'; ``` To enable binary logging, you will need to update your RDS instance's parameter group. To create a new parameter group: navigate to `RDS` > `Parameter groups` > click `Create`. For Aurora clusters, please ensure that you are creating this parameter group as **DB cluster parameter group**. 1. Create the parameter group. 2. Click into the parameter group and modify `binlog_format` to `ROW` 3. Restart your database. ![Create parameter group](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/mysql/create_parameter_group.png) ![Edit binlog format](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/mysql/edit_binlog_format.png) The default value for binlog retention hours is NULL, which means they are not retained. We recommend setting this to a value that is at least 24 hours. ```sql -- If you are using AWS RDS or Aurora CALL mysql.rds_set_configuration('binlog retention hours', 24); -- If you are using Azure SET GLOBAL binlog_expire_logs_seconds = 604800; -- You do not need to set this for GCP CloudSQL or generic MySQL ``` ### Additional features * GTID support * Automatic support for [gh-ost migrations](https://github.com/github/gh-ost) Please make sure the following settings are enabled: * gtid\_mode = ON * enforce\_gtid\_consistency = ON Once this is configured, Artie will automatically use GTID to replicate. # Amazon RDS Source: https://artie.com/docs/sources/oracle/amazon-rds ## Required settings * Host name * Port (default is `1521`) * Service account * Database name * Schema name * Database log mode to be `ARCHIVELOG` * Supplemental logging to be enabled Due to the [limitation of Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Concepts.CDBs.html#Oracle.Concepts.single-tenant-limitations), we cannot replicate from a multi-tenant database. ## Setup instructions To enable archive log mode, you'll need to enable automated backups. This change will take effect upon the next database restart. Once this is enabled, we recommend you to set the log retention to at least 24 hours. ```sql -- We recommend setting the log retention to at least 24 hours. EXECUTE rdsadmin.rdsadmin_util.set_configuration('archivelog retention hours', 24); COMMIT; ``` ```sql EXECUTE rdsadmin.rdsadmin_util.alter_supplemental_logging(p_action => 'ADD'); COMMIT; ``` ```sql CREATE USER ARTIE_USER IDENTIFIED BY PASSWORD; CREATE TABLESPACE ARTIE_LOGMINER_TBS DATAFILE SIZE 25M AUTOEXTEND ON MAXSIZE UNLIMITED; ALTER USER ARTIE_USER DEFAULT TABLESPACE ARTIE_LOGMINER_TBS; ALTER USER ARTIE_USER QUOTA UNLIMITED ON ARTIE_LOGMINER_TBS; GRANT CREATE SESSION TO ARTIE_USER; -- Grant access to LogMiner GRANT LOGMINING TO ARTIE_USER; -- Flashback for backfilling GRANT FLASHBACK ANY TABLE TO ARTIE_USER; GRANT SELECT ANY TRANSACTION TO ARTIE_USER; -- Required to read and write to the data dictionary to track schema history GRANT SELECT_CATALOG_ROLE TO ARTIE_USER; GRANT EXECUTE_CATALOG_ROLE TO ARTIE_USER; -- We'll need this to create a table for managing internal log buffers (LGWR) GRANT CREATE TABLE TO ARTIE_USER; GRANT CREATE SEQUENCE TO ARTIE_USER; -- Read access to system tables (redo, archive and current transaction logs) EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$DATABASE','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$LOG','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$LOG_HISTORY','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGMNR_LOGS','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGMNR_CONTENTS','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGMNR_PARAMETERS','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGFILE','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$ARCHIVED_LOG','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$ARCHIVE_DEST_STATUS','ARTIE_USER','SELECT'); EXECUTE rdsadmin.rdsadmin_util.grant_sys_object('V_$TRANSACTION','ARTIE_USER','SELECT'); -- Grant access to the service account to query the tables. -- GRANT SELECT ON {schema}.{table} TO ARTIE_USER; GRANT SELECT ANY TABLE TO ARTIE_USER; ``` # PostgreSQL Source: https://artie.com/docs/sources/postgresql ## Required settings * Host name * Port (default is `5432`) * Service account * Database name (with logical replication enabled) ## Provider specific instructions ## Additional features PostgreSQL 16 introduces the ability to use logical replication on a standby server. Steps to enable this: 1. Create `dbz_publication` publication on the primary database ```sql CREATE PUBLICATION dbz_publication FOR ALL TABLES; ``` 2. Enable `hot_standby_feedback` on the standby server ```sql postgres> SHOW hot_standby_feedback; hot_standby_feedback ---------------------- on ``` 3. `WAL_LEVEL` is set to `logical` on the standby server ```sql postgres> SHOW WAL_LEVEL; wal_level ---------------------- logical ``` Once you selected `Enable heartbeats` under Advanced settings, you will then need to run the following command: ```sql CREATE TABLE test_heartbeat_table (id text PRIMARY KEY, ts timestamp); -- Grant access to the heartbeat table GRANT UPDATE ON TABLE test_heartbeat_table TO artie_transfer; -- Then insert one row into this table. -- Artie's periodic pings will be this: -- UPDATE test_heartbeat_table set ts = now() WHERE id = '1'; -- Such that we never end up adding additional rows. INSERT INTO test_heartbeat_table (id, ts) VALUES (1, NOW()); ``` By default, Artie will create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) that includes all table changes. You can override this behavior by selecting `Filtered` under the Deployment advanced settings. TOAST (The Oversized-Attribute Storage Technique) is a mechanism used by Postgres to store large column values to alleviate the page size limit. TOAST columns may be different data types such as `text`, `jsonb`, `bytea`, `array`, etc. One thing that is special about TOAST columns is that if the column value has not changed, the column value will not be included in WAL if the table's replica identity is set to `default`. Artie has automatic detection built-in to detect an unchanged TOAST column and will conditionally replicate this into the target database. Our built-in process will detect if an incoming CDC event has an unchanged TOAST column and will not update the column value in the target database. We are able to do all of this without having to increase your table replica identity. We have regular monitors that provide additional guardrails around your database replication, and will do the following: * **Regularly check and monitor** your replication slot size in 15-minute intervals and notify if the slot exceeds a certain threshold. * **Heartbeats verification.** For folks that are leveraging heartbeats, we will alert you if a heartbeat query fails. * **Terminate any idle queries** that are lingering for more than a day. You can avoid having this problem by setting `idle_in_transaction_session_timeout`. # Amazon Aurora and RDS Source: https://artie.com/docs/sources/postgresql/amazon-rds ## Creating a service account ```sql CREATE USER artie_transfer WITH PASSWORD 'password'; -- (optional) If the schema is not public, you will need this additional line GRANT USAGE ON SCHEMA schema_name TO artie_transfer; -- Grant read-only access to future tables ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name GRANT SELECT ON TABLES TO artie_transfer; -- Grant access to existing tables GRANT SELECT ON ALL TABLES IN SCHEMA schema_name TO artie_transfer; -- The replication role does not have enough permissions to create publications. -- So you will need to create this as well. CREATE PUBLICATION dbz_publication FOR ALL TABLES; -- Grant replication role GRANT rds_replication to artie_transfer; ``` ## Turning on logical replication **rds.logical\_replication** is a [static](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Parameters.html) parameter which requires a database reboot to take effect. 1. Select your database instance 2. Go into `Configuration` and find your `DB instance parameter group` 3. If you don't have one, go to `Parameter groups` and click `Create parameter group` ![Create parameter group](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/create-parameter-group.png) 4. Find and change `rds.logical_replication` and set it to `1` ![Change rds.logical\_replication to 1](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/rds.logical_replication.png) 5. Associate the parameter group with your database instance ![Associate parameter group to your database instance](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/associate-rds-parameter-group.png) # GCP CloudSQL Source: https://artie.com/docs/sources/postgresql/gcp-cloudsql ## Creating a service account ```sql CREATE USER artie_transfer WITH PASSWORD 'password'; -- (optional) If the schema is not public, you will need this additional line GRANT USAGE ON SCHEMA schema_name TO artie_transfer; -- Grant read-only access to future tables ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name GRANT SELECT ON TABLES TO artie_transfer; -- Grant access to existing tables GRANT SELECT ON ALL TABLES IN SCHEMA schema_name TO artie_transfer; -- The replication role does not have enough permissions to create publications. -- So you will need to create this as well. CREATE PUBLICATION dbz_publication FOR ALL TABLES; ALTER USER artie_transfer REPLICATION; ``` ## Turning on Logical Decoding Flag Steps to turn on logical decoding: 1. Find your database instance and click `Edit` 2. Navigate to `Flags` and find and set `cloudsql.logical_decoding` to `On` ![CloudSQL Logical Decoding](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/cloudsql_enable_logical_decoding.png) # Supabase Source: https://artie.com/docs/sources/postgresql/supabase ## 1. Enable IPv4 Add On To do this: 1. Go to your project settings in Supabase 2. Click on Add Ons 3. Enable IPv4 ![Supabase IPv4 Add on](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/supabase_ipv4.png) ## 2. Find your Supabase credentials You can find this under Settings > Database > Connection Parameters. Be sure to uncheck connection pooler! We will need the following: 1. Database Host 2. Database Port (Should be 5432) 3. Username 4. Password 5. Database To create a service account, run this in Supabase: ```sql CREATE USER artie_transfer WITH PASSWORD 'password'; -- Grant replication permissions ALTER USER artie_transfer REPLICATION; -- Grant read-only access to future tables ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO artie_transfer; -- Grant access to existing tables GRANT SELECT ON ALL TABLES IN SCHEMA public TO artie_transfer; CREATE PUBLICATION dbz_publication FOR ALL TABLES; ``` # Advanced Settings Source: https://artie.com/docs/tables/advanced-settings To find table settings, click on your deployment > `Edit` > `Tables` > `Table settings` If your BigQuery table is partitioned, you can specify the partitioned columns here. Click [here](/guides/database/enable-bigquery-partitioning) for instructions on how to partition your table in BigQuery. Artie merges data into your Snowflake based on your source table's primary key(s). If you have an additional cluster key defined, you can specify that here. For example, if the table was clustered by `created_at`, you can specify that column here. ```sql ALTER TABLE foo CLUSTER BY (TO_DATE(created_at)) ``` This setting allows you to define a new table alias. If your source table is named `foo`, you can name it `bar`. ![Table alias](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/table_alias.png) When enabled, Artie will skip processing deletes. This setting is useful if you want to keep a smaller dataset in your source and have a complete archive in your destination. For high volume tables, Artie also has an ability to specify a different set of flush rules. We will automatically do this for you based on your ingestion lag. ## History tables With history table, Artie will create a separate table that stores all data changes over time, along with the timestamp and operation timestamp. Click [here](https://www.artie.com/blogs/history-table) to learn more! **Steps to enable history mode:** 1. Edit your deployment 2. Go into the `Tables` tab 3. Find the table you want to enable history mode ![History table](https://mintlify.s3.us-west-1.amazonaws.com/artie/assets/history_table.png)