Re-Air: Confidently Wrong: Why AI Needs Tools (and So Do We)

Dec 3, 2025

In this insightful discussion, Matt, known as the 'Cynical Data Guy', shares his experience with the evolution of AI, particularly GPT-5, noting its modest improvements and persistent issues. He delves into the risks of relying on tools, highlighting the need for control layers to mitigate AI hallucinations. The conversation addresses the ongoing debate of data warehouses versus agent-based approaches, and how understanding risk through poker can inform data teams. Matt also recommends reading fiction to enhance comprehension of human behavior in data work.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

INSIGHT

Tool-First Models Reduce Hallucinations

Open-source LLMs that hallucinate can become useful when designed to rely on external tools.
Tool-first models trade internal factual certainty for deterministic tool calls to reduce harmful hallucinations.

ADVICE

Combine LLMs With Deterministic Tools

Design systems where the LLM acts as an interface and deterministic tools handle critical operations.
Combine probabilistic models with deterministic tooling to retain control and reliability.

INSIGHT

Data Warehouses Versus Agent-Based Access

The future battle in data will be

Get the Snipd Podcast app to discover more snips from this episode

Get the app

This episode is a re-air of one of our most popular conversations from this year, featuring insights worth revisiting. Thank you for being part of the Data Stack community. Stay up to date with the latest episodes at datastackshow.com.

This week on The Data Stack Show, John and Matt dive into the latest trends in AI, discussing the evolution of GPT models, the role of tools in reducing hallucinations, and the ongoing debate between data warehouses and agent-based approaches. They also explore the complexities of risk-taking in data teams, drawing lessons from Nate Silver’s book on risk and sharing real-world analogies from cybersecurity, football, and political campaigns. Key takeaways include the importance of balancing innovation with practical risk management, the need for clear recommendations from data professionals, the value of reading fiction to understand human behavior in data, and so much more.

Highlights from this week’s conversation include:

Initial Impressions of GPT-5 (1:41)
AI Hallucinations and the Open-Source GPT Model (4:06)
Tools and Determinism in AI Agents (6:00)
Risks of Tool Reliance in AI (8:05)
The Next Big Data Fight: Warehouses vs. Agents (10:21)
Real-Time Data Processing Limitations (12:56)
Risk in Data and AI: Book Recommendation (17:08)
Measurable vs. Perceived Risk in Business (20:10)
Security Trade-Offs and Organizational Impact (22:31)
The Quest for Certainty and Wicked Learning Environments (27:37)
Poker, Process, and Data Team Longevity (29:11)
Support Roles and Limits of Data Teams (32:56)
Final Thoughts and Takeaways (34:20)

The Data Stack Show is a weekly podcast powered by RudderStack, customer data infrastructure that enables you to deliver real-time customer event data everywhere it’s needed to power smarter decisions and better customer experiences. Each week, we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.