Alter Everything 194: AI and Data Pipelines
9 snips
Sep 24, 2025 Nick Schrock, CTO and founder of Dagster Labs, shares insights from his extensive background in developer tools and GraphQL. He discusses the crucial aspects of AI data readiness, emphasizing the importance of metadata and context engineering. Schrock highlights the challenges of context rot and the need for governance in AI workflows. He also provides strategies for balancing speed and quality in data pipelines, while advising teams on grounding AI mandates in real value. With enthusiasm, he predicts a bright future for data engineering, aided by AI.
AI Snips
Chapters
Transcript
Episode notes
Data Readiness Is Unchanged
- Good data practices for humans remain the same for AI; models succeed when schemas and documentation are clear.
- Nick Schrock calls giving the right context to models 'context engineering' and says it's an engineering problem.
Control Context, Avoid Context Rot
- Selectively query high-quality context instead of dumping everything into the model.
- Limit context window size and avoid context rot by keeping prompts targeted and iterative.
Sandbox AI Changes And Treat Context As Code
- Compartmentalize AI changes into small, well-defined components so you can govern and rollback easily.
- Treat context corrections and prompt rules as code with versioning and rollbacks.