In this episode of "How AI is Built", data architect Anjan Banerjee provides an in-depth look at the world of data architecture and building complex AI and data systems. Anjan breaks down the basics using simple analogies, explaining how data architecture involves sorting, cleaning, and painting a picture with data, much like organizing Lego bricks to build a structure.
Summary by Section
Introduction
- Anjan Banerjee, a data architect, discusses building complex AI and data systems
- Explains the basics of data architecture using Lego and chat app examples
Sources and Tools
- Identifying data sources is the first step in designing a data architecture
- Pick the right tools to extract data based on use cases (block storage for images, time series DB, etc.)
- Use one tool for most activities if possible, but specialized tools offer benefits
- Multi-modal storage engines are gaining popularity (Snowflake, Databricks, BigQuery)
Airflow and Orchestration
- Airflow is versatile but has a learning curve; good for orgs with Python/data engineering skills
- For less technical orgs, GUI-based tools like Talend, Alteryx may be better
- AWS Step Functions and managed Airflow are improving native orchestration capabilities
- For multi-cloud, prefer platform-agnostic tools like Astronomer, Prefect, Airbyte
AI and Data Processing
- ML is key for data-intensive use cases to avoid storing/processing petabytes in cloud
- TinyML and edge computing enable ML inference on device (drones, manufacturing)
- Cloud batch processing still dominates for user targeting, recommendations
Data Lakes and Storage
- Storage choice depends on data types, use cases, cloud ecosystem
- Delta Lake excels at data versioning and consistency; Iceberg at partitioning and metadata
- Pulling data into separate system often needed for advanced analytics beyond source system
Data Quality and Standardization
- "Poka-yoke" error-proofing of input screens is vital for downstream data quality
- Impose data quality rules and unified schemas (e.g. UTC timestamps) during ingestion
- Complexity arises with multi-region compliance (GDPR, CCPA) requiring encryption, sanitization
Hot Takes and Wishes
- Snowflake is overhyped; great UX but costly at scale. Databricks is preferred.
- Automated data set joining and entity resolution across systems would be a game-changer
Anjan Banerjee:
Nicolay Gerold:
00:00 Understanding Data Architecture
12:36 Choosing the Right Tools
20:36 The Benefits of Serverless Functions
21:34 Integrating AI in Data Acquisition
24:31 The Trend Towards Single Node Engines
26:51 Choosing the Right Database Management System and Storage
29:45 Adding Additional Storage Components
32:35 Reducing Human Errors for Better Data Quality
39:07 Overhyped and Underutilized Tools
Data architecture, AI, data systems, data sources, data extraction, data storage, multi-modal storage engines, data orchestration, Airflow, edge computing, batch processing, data lakes, Delta Lake, Iceberg, data quality, standardization, poka-yoke, compliance, entity resolution
