Designing and Evaluating Agent Tools

Summary
In this episode of the AI Engineering Podcast Lucas Thelosen and Drew Gillson talk about Orion, their agentic analytics platform that delivers proactive, push-based insights to business users through asynchronous thinking with rich organizational context. Lucas and Drew share their approach to building trustworthy analysis by grounding in semantic layers, fact tables, and quality-assurance loops, as well as their focus on accuracy through parallel test-time compute and evolving from probabilistic steps to deterministic tools. They discuss the importance of context engineering, multi-agent orchestration, and security boundaries for enterprise deployments, and share lessons learned on consistency, tool design, user change management, and the emerging role of "AI manager" as a career path. The conversation highlights the future of AI knowledge workers collaborating across organizations and tools while simplifying UIs and raising the bar on actionable, trustworthy analytics.

Announcements

• Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems
• When ML teams try to run complex workflows through traditional orchestration tools, they hit walls. Cash App discovered this with their fraud detection models - they needed flexible compute, isolated environments, and seamless data exchange between workflows, but their existing tools couldn't deliver. That's why Cash App rely on Prefect. Now their ML workflows run on whatever infrastructure each model needs across Google Cloud, AWS, and Databricks. Custom packages stay isolated. Model outputs flow seamlessly between workflows. Companies like Whoop and 1Password also trust Prefect for their critical workflows. But Prefect didn't stop there. They just launched FastMCP - production-ready infrastructure for AI tools. You get Prefect's orchestration plus instant OAuth, serverless scaling, and blazing-fast Python execution. Deploy your AI tools once, connect to Claude, Cursor, or any MCP client. No more building auth flows or managing servers. Prefect orchestrates your ML pipeline. FastMCP handles your AI tool infrastructure. See what Prefect and Fast MCP can do for your AI workflows at aiengineeringpodcast.com/prefect today.
• Your host is Tobias Macey and today I'm interviewing Lucas Thelosen and Drew Gillson about their experiences building an agentic analytics platform and the challenges of ensuring accuracy to build trust

Interview

• Introduction
• How did you get involved in machine learning?
• Can you describe what Orion is and the story behind it?
• Business analytics is a field that requires a high degree of accuracy and detail because of the potential for substantial impact on the business (positive and negative). These are areas that generative AI has struggled with achieving consistently. What was your process for building confidence in your ability to achieve that threshold before committing to the path you are on now?
• There are numerous ways that generative AI can be incorporated into the process of designing, building, and delivering analytical insights. How would you characterize the different strategies that data teams and vendors have approached that problem?
  • What do you see as the organizational benefits of moving to a push-based model for analytics?
• Can you describe the system architecture of Orion?
• Agentic design patterns are still in the early days of being developed and proven out. Can you give a breakdown of the approach that you are using?
• How do you think about the responsibility boundaries, communication paths, temporal patterns, etc. across the different agents?
  • Tool use is a key component of agentic architectures. What is your process for identifying, developing, validating, and securing the tools that you provide to your agents?
• What are the boundaries and extension points that you see when building agentic systems? 
  • What are the opportunities for using e.g. A2A for protocol for managing agentic hand-offs?
• What is your process for managing the experimentation loop for changes to your models, data, prompts, etc. as you iterate on your product?
• What are some of the ways that you are using the agents that power your system to identify and act on opportunities for self-improvement?
• What are the most interesting, innovative, or unexpected ways that you have seen Orion used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Orion?
• When is an agentic approach the wrong choice?
• What do you have planned for the future of Orion?

Contact Info

• Lucas
  • LinkedIn
• Drew
  • LinkedIn

Parting Question

• From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email hosts@aiengineeringpodcast.com with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers.

Links

• Gravity
• Orion 
  • Data Engineering Podcast Episode
• Site Reliability Engineering
• Anthropic Claude Sonnet 4.5
• A2A (Agent2Agent) Protocol
• Simon Willison
  • AI Lethal Trifecta
• Behavioral Science
• Grounded Theory
• LLM as a Judge
• RLHF == Reinforcement Learning from Human Feedback

The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0