Agent Bricks and How Data-AI Integration Changes Everything with Craig Wiley

Sep 26, 2025

Craig Wiley, Senior Director of Product Management at Databricks, leads the conversation on the crucial link between data and AI in accelerating ML development. He shares eye-opening statistics: 85% of AI evaluations are done solely by engineers. Craig dives into the evolution from classical ML to GenAI, revealing tools like Agent Bricks for better evaluations. Customer success stories from Suncorp and AstraZeneca illustrate the real-world impact of Mosaic AI. He champions the need for trustable and systematic model evaluations in enterprise AI.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

From Amazon To Building SageMaker

Craig Wiley started in Amazon product roles and moved into ML by solving analyst speed problems with engineers.
That work led him to help build SageMaker and later to Databricks to connect data and AI more tightly.

INSIGHT

Data+AI Connection Multiplies Speed

Connecting the AI layer to the data layer can deliver an order-of-magnitude acceleration in development cycles.
Craig joined Databricks to exploit that connection and speed up ML development.

INSIGHT

Mosaic Unifies End-To-End AI Workflows

Mosaic AI bundles Databricks' ML and AI capabilities into a single brand to simplify customer adoption.
Databricks' platform spans from data ingestion through analytics, training, and deployment, emphasizing end-to-end lineage.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

"85% of AI use cases are being evaluated by the engineer who built it saying, 'yep, seemed to work pretty well.' If you're gonna build a system that's going to be critical to the business, that's going to be important that it gets it right, then you can't do that without evaluations." - Craig Wiley

Fresh out of the studio, Craig Wiley, Senior Director of Product Management at Databricks who leads Mosaic AI, joins us to discuss the forefront of enterprise AI from model development to deployment at scale. Beginning with his career journey in ML operations, Craig explained how he recognized the critical connection between data and AI layers that could deliver order-of-magnitude acceleration in development cycles. Emphasizing the transition from classical ML operations to LLM operations, he showcased how Databricks' unified platform eliminates training-serving skew through data lineage capabilities and supports both fine-tuning and RAG approaches depending on industrial use case requirements. Highlighting compelling customer success stories including Suncorp's employee productivity platform and AstraZeneca's transformation of 400,000 clinical trial documents into queryable insights, Craig revealed a striking reality about enterprise AI evaluation - that 85% of AI use cases are being evaluated only by the engineers who built them, reinforcing that proper evaluation frameworks remain foundational for trustworthy AI implementation. He concluded by introducing Agent Bricks as Databricks' evaluation-centric approach to building production agents, emphasizing that model flexibility and rigorous testing are essential for enterprises moving from experimentation to production, while sharing his vision that the industry must evolve from the "year of agents" to the "year of evaluation and quality."

Episode Highlights: [00:00] Quote of the Day by Craig Wiley [01:21] How Craig Wiley started his work in ML Ops that led him to Databricks [02:43] Data and AI layer connection creates order-of-magnitude acceleration [03:47] Mosaic AI acquisition expanded Gen AI solution capabilities [04:38] Classical ML statistics versus Gen AI evaluation challenges [05:48] Mosaic AI covers end-to-end from data ingestion [07:12] Training-serving skew eliminated through unified platform lineage [08:51] Fine tuning versus RAG depends on use case [10:49] Industrial agents benefit from fine-tuned smaller models [12:44] Common governance scheme covers tables through model access [13:52] Agent Bricks prioritizes accuracy over simplicity alone [15:44] Model flexibility crucial for speed and accuracy optimization [16:54] AB testing different models shows immediate performance differences [17:59] Suncorp and AstraZeneca demonstrate diverse AI applications [19:37] Asia Pacific shows aggressive AI adoption strategies [20:59] CFO approval requires proven agent effectiveness evaluation [22:00] 85% of AI cases evaluated only by building engineer [23:20] Model agnostic approach beats single-vendor AI strategies [24:12] Industry terminology evolves rapidly from RAG to agents [25:39] Customer creativity with governance capabilities inspires product development

Profile: Craig Wiley, Senior Director of Product Management at Databricks and Mosaic AI

LinkedIn: https://www.linkedin.com/in/craigwiley/

Podcast Information: Bernard Leong hosts and produces the show. The proper credits for the intro and end music are "Energetic Sports Drive." G. Thomas Craig mixed and edited the episode in both video and audio format.

Here are the links to watch or listen to our podcast.

Analyse Asia Main Site: https://analyse.asia

Analyse Asia Spotify: https://open.spotify.com/show/1kkRwzRZa4JCICr2vm0vGl

Analyse Asia Apple Podcasts: https://podcasts.apple.com/us/podcast/analyse-asia-with-bernard-leong/id914868245

Analyse Asia LinkedIn: https://www.linkedin.com/company/analyse-asia/

Analyse Asia X (formerly known as Twitter): https://twitter.com/analyseasia

Subscribe Newsletter on LinkedIn https://www.linkedin.com/build-relation/newsletter-follow?entityUrn=7149559878934540288