> ## Documentation Index
> Fetch the complete documentation index at: https://artie.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Artie's typing library

> Learn how Artie's internal typing library ensures data integrity and schema evolution across your data sources.

## Overview 🎯

Artie's custom [typing library](https://github.com/artie-labs/transfer/tree/master/lib/typing) is the backbone of our data transfer system. It ensures that your data maintains its original structure and types when moving between sources and destinations.

### Key Features ✨

* **Schema Evolution**: Automatically detects and adds missing columns
* **Type Inference**: Intelligently determines data types from values
* **Performance**: 2x faster than Go's Reflect library
* **Data Integrity**: Zero data transformation or loss

## Schema Propagation Behavior

Artie's typing library governs when schema changes are picked up and reflected downstream - which depends on whether your source is relational or non-relational.

### Relational Sources (e.g., Postgres, MySQL)

* Artie reads the source schema directly
* New tables and columns are synced immediately, even when values are all NULL
* No need to wait for populated data - schema updates are applied on first detection

### Non-Relational Sources (e.g., DynamoDB, MongoDB)

Here, schemas are not strictly defined, so Artie takes a more cautious approach:

* New columns are typed based on the first not-NULL value
* If a new column or table only contains NULLs, it won’t be propagated until real data shows up
* This helps avoid guessing or applying incorrect types based on incomplete information

> 💡 **Pro Tip**: For non-relational sources, insert a row with actual values to trigger schema propagation downstream - Artie will inspect the data type and infer the correct type in the destination.

## Data Type Handling 🔍

Artie's typing library intelligently handles data type mapping between your source and destination systems. Here's how it works:

1. **Initial Type Detection**: When data first arrives, Artie analyzes the source schema and values to determine the most appropriate data type for your destination.
2. **Type Preservation**: Once a data type is established in your destination, Artie uses the destination schema as a source of truth and maintains that type to ensure consistency.
3. **Data Type Conversion**: If source and destination types differ, Artie automatically converts the data while preserving its integrity.

<Accordion title="What happens if type conversion fails?">
  If Artie encounters a type conversion that could lead to data loss or errors, we'll:

  * Raise a clear error message
  * Notify you immediately
  * Work with you to resolve the issue
</Accordion>

### Numbers

We preserve the exact numeric type from your source (Postgres, MySQL, or MongoDB). Here's how we handle different number formats:

* `5` → Integer
* `"5"` → String
* `5.0` → Float

### JSON Objects

We maintain JSON objects exactly as they appear in your source, with no flattening or transformation.

<Frame captions="JSON object preservation in Snowflake example">
  <img src="https://mintcdn.com/artie/cR74rDu7gj_LCvTI/assets/snowflake_json_obj.png?fit=max&auto=format&n=cR74rDu7gj_LCvTI&q=85&s=b3d5ad5e70b7c74b434b15defcee3798" alt="" width="772" height="369" data-path="assets/snowflake_json_obj.png" />
</Frame>

> 💡 **Pro Tip**: If you prefer to store JSON as a string, you can explicitly specify the column type as `STRING` in your schema.

### Arrays

We support both simple and complex array structures:

* Simple arrays
* Nested arrays with objects

<img src="https://mintcdn.com/artie/cR74rDu7gj_LCvTI/assets/snowflake_array.png?fit=max&auto=format&n=cR74rDu7gj_LCvTI&q=85&s=27a5428fa9296a44a2aaab75f8a933dc" alt="Normal array" width="779" height="489" data-path="assets/snowflake_array.png" />

<img src="https://mintcdn.com/artie/cR74rDu7gj_LCvTI/assets/snowflake_array_nested_objects.png?fit=max&auto=format&n=cR74rDu7gj_LCvTI&q=85&s=0574e96ea15996adea81f50b24809453" alt="Array with nested objects" width="819" height="414" data-path="assets/snowflake_array_nested_objects.png" />

### Timestamps, Dates, and Times ⏰

Our library supports 15+ different time formats with zero precision loss. We preserve your original time layout using our custom `time.Time` object.

> 💡 **Pro Tip**: Like JSON objects, you can override time type inference by specifying your preferred data type in the schema.

## Advanced

<Accordion title="Postgres variable numeric">
  A <Tooltip tip="A variable numeric column is a numeric data type without precision or scale specified. ">variable numeric column</Tooltip> in Postgres requires special handling when transferring to different destinations. By default, we will create this type as a string equivalent column so that we do not have to trade off data precision. You can override this behavior by modifying the destination column type.

  > Why do we treat a variable numeric column differently?

  We treat this column type differently because Postgres will allow you to *store arbitrary amount of digits* in each cell up to the <Tooltip tip="Up to 131072 digits before the decimal point; up to 16383 digits after the decimal point">implementation limit</Tooltip>. This exceeds most destination's numeric data type limits, as such we will create a string equivalent column to preserve data precision.
</Accordion>

## Need Help? 🤝

If you have questions not covered here, reach out to us at [hi@artie.com](mailto:hi@artie.com)!