258: Confidently Wrong: Why AI Needs Tools (and So Do We)
Aug 20, 2025
This discussion dives into AI's latest advancements, especially the evolution of GPT models and the importance of reliable tools to minimize hallucinations. It contrasts data warehouses with agent-based approaches, addressing the risks involved. The conversation features engaging analogies from poker and football, emphasizing the art of risk-taking in data teams. Notably, the hosts advocate for balancing innovation with risk management and highlight the value of fiction in understanding human behavior—all while navigating the complexities of real-time data processing.
35:18
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
insights INSIGHT
Tool-First Models Reduce Hallucinations
Open-source LLMs may hallucinate frequently but are designed to call deterministic tools to compensate.
Tool-first models trade internal factuality for external determinism to improve real-world usefulness.
insights INSIGHT
LLMs As Interfaces To Deterministic Tools
Combining LLMs with deterministic tools makes agents more reliable and controllable.
The LLM becomes an interface while the tools perform precise, deterministic work.
insights INSIGHT
No One-Size Data Architecture
The warehouse vs agent debate will hinge on use case, data volume, and team preference.
Large-scale, complex joins and transformations still favor centralized warehouses or lakehouses today.
Get the Snipd Podcast app to discover more snips from this episode
This week on The Data Stack Show, John and Matt dive into the latest trends in AI, discussing the evolution of GPT models, the role of tools in reducing hallucinations, and the ongoing debate between data warehouses and agent-based approaches. They also explore the complexities of risk-taking in data teams, drawing lessons from Nate Silver’s book on risk and sharing real-world analogies from cybersecurity, football, and political campaigns. Key takeaways include the importance of balancing innovation with practical risk management, the need for clear recommendations from data professionals, the value of reading fiction to understand human behavior in data, and so much more.
Highlights from this week’s conversation include:
Initial Impressions of GPT-5 (1:41)
AI Hallucinations and the Open-Source GPT Model (4:06)
Tools and Determinism in AI Agents (6:00)
Risks of Tool Reliance in AI (8:05)
The Next Big Data Fight: Warehouses vs. Agents (10:21)
Real-Time Data Processing Limitations (12:56)
Risk in Data and AI: Book Recommendation (17:08)
Measurable vs. Perceived Risk in Business (20:10)
Security Trade-Offs and Organizational Impact (22:31)
The Quest for Certainty and Wicked Learning Environments (27:37)
Poker, Process, and Data Team Longevity (29:11)
Support Roles and Limits of Data Teams (32:56)
Final Thoughts and Takeaways (34:20)
The Data Stack Show is a weekly podcast powered by RudderStack, customer data infrastructure that enables you to deliver real-time customer event data everywhere it’s needed to power smarter decisions and better customer experiences. Each week, we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.
RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.
The Data Stack Show