The Data Stack Show 175: The Parts, Pieces, and Future of Composable Data Systems, Featuring Wes McKinney, Pedro Pedreira, Chris Riccomini, and Ryan Blue
17 snips
Jan 31, 2024 Data systems experts Wes McKinney, Pedro Pedreira, Chris Riccomini, and Ryan Blue discuss the concept of composable data systems, the challenges and incentives for composable components, specialization and modularity in data workloads, and the efficiency and common layers in data management systems. They also explore the evolution of data system composability, exciting new projects in data systems, and the challenges of standardizing APIs.
AI Snips
Chapters
Transcript
Episode notes
Push Policy Into Storage Layer
- Storage-layer standards like table formats must carry policy and metadata to make multi-engine use safe.
- Moving access controls and catalog decisions toward storage helps keep policies coherent across engines.
Schema Friction Breaks Systems
- Data model and schema description span from runtime to file formats and cause much coercion.
- A shared, expressive schema standard would reduce repeated type conversions across systems.
Type Systems Force Trade-offs
- Different type-system goals cause fragmentation: expressive in-memory types versus minimal portable storage types.
- Projects must balance implementability versus expressiveness when standardizing types.