Episode 49: Why Data and AI Still Break at Scale (and What to Do About It)

Jun 5, 2025

Akshay Agrawal, founder of Marimo and former Google Brain researcher, discusses the critical challenges faced in AI at scale. He emphasizes the need for robust infrastructure over just improved models. The conversation covers the importance of reproducibility and the shortcomings of traditional tools. Akshay introduces Marimo's innovative design that addresses modular AI applications and the difficulties in debugging large language models. Live demos illustrate Marimo's capabilities in data extraction and agentic workflows, merging technical insights with cultural reflections in data science.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Declarative Data-Centric Paradigm

The future of ML tools trends towards declarative, data-centric functional programming. - This shift enables better optimization, reproducibility, and developer experience.

ANECDOTE

Hidden State Bug Example

Deleting a code cell in traditional notebooks can leave variables in memory, causing silent bugs. - Marimo automatically detects dependencies and invalidates downstream code when cells are deleted.

ADVICE

Leverage Marimo's Versatility

Use Marimo for interactive exploration, prototyping, and data apps that can run as scripts. - Leverage Marimo's features to blur the lines between notebooks and scripts for better reproducibility.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

If we want AI systems that actually work in production, we need better infrastructure—not just better models.

In this episode, Hugo talks with Akshay Agrawal (Marimo, ex-Google Brain, Netflix, Stanford) about why data and AI pipelines still break down at scale, and how we can fix the fundamentals: reproducibility, composability, and reliable execution.

They discuss:
🔁 Why reactive execution matters—and how current tools fall short
🛠️ The design goals behind Marimo, a new kind of Python notebook
⚙️ The hidden costs of traditional workflows (and what breaks at scale)
📦 What it takes to build modular, maintainable AI apps
🧪 Why debugging LLM systems is so hard—and what better tooling looks like
🌍 What we can learn from decades of tools built for and by data practitioners

Toward the end of the episode, Hugo and Akshay walk through two live demos: Hugo shares how he’s been using Marimo to prototype an app that extracts structured data from world leader bios, and Akshay shows how Marimo handles agentic workflows with memory and tool use—built entirely in a notebook.

This episode is about tools, but it’s also about culture. If you’ve ever hit a wall with your current stack—or felt like your tools were working against you—this one’s for you.

LINKS

🎓 Want to go deeper?
Check out Hugo's course: Building LLM Applications for Data Scientists and Software Engineers.
Learn how to design, test, and deploy production-grade LLM systems — with observability, feedback loops, and structure built in.
This isn’t about vibes or fragile agents. It’s about making LLMs reliable, testable, and actually useful.

Includes over $800 in compute credits and guest lectures from experts at DeepMind, Moderna, and more.
Cohort starts July 8 — Use this link for a 10% discount