Artie's Typing Library
Learn how Artie’s internal typing library ensures data integrity and schema evolution across your data sources.
Overview 🎯
Artie’s custom typing library is the backbone of our data transfer system. It ensures that your data maintains its original structure and types when moving between sources and destinations.
Key Features ✨
- Schema Evolution: Automatically detects and adds missing columns
- Type Inference: Intelligently determines data types from values
- Performance: 2x faster than Go’s Reflect library
- Data Integrity: Zero data transformation or loss
Data Type Handling 🔍
Artie’s typing library intelligently handles data type mapping between your source and destination systems. Here’s how it works:
- Initial Type Detection: When data first arrives, Artie analyzes the source schema and values to determine the most appropriate data type for your destination.
- Type Preservation: Once a data type is established in your destination, Artie maintains that type to ensure consistency.
- Automatic Conversion: If source and destination types differ, Artie automatically converts the data while preserving its integrity.
Numbers
We preserve the exact numeric type from your source (Postgres, MySQL, or MongoDB). Here’s how we handle different number formats:
5
→ Integer"5"
→ String5.0
→ Float
JSON Objects
We maintain JSON objects exactly as they appear in your source, with no flattening or transformation.
💡 Pro Tip: If you prefer to store JSON as a string, you can explicitly specify the column type as
STRING
in your schema.
Arrays
We support both simple and complex array structures:
- Simple arrays
- Nested arrays with objects
Timestamps, Dates, and Times ⏰
Our library supports 15+ different time formats with zero precision loss. We preserve your original time layout using our custom time.Time
object.
💡 Pro Tip: Like JSON objects, you can override time type inference by specifying your preferred data type in the schema.
Need Help? 🤝
If you have questions not covered here, reach out to us at [email protected]!
Was this page helpful?