Importance of model flexibility and A/B testing

"85% of AI use cases are being evaluated by the engineer who built it saying, 'yep, seemed to work pretty well.' If you're gonna build a system that's going to be critical to the business, that's going to be important that it gets it right, then you can't do that without evaluations." - Craig Wiley

Fresh out of the studio, Craig Wiley, Senior Director of Product Management at Databricks who leads Mosaic AI, joins us to discuss the forefront of enterprise AI from model development to deployment at scale. Beginning with his career journey in ML operations, Craig explained how he recognized the critical connection between data and AI layers that could deliver order-of-magnitude acceleration in development cycles. Emphasizing the transition from classical ML operations to LLM operations, he showcased how Databricks' unified platform eliminates training-serving skew through data lineage capabilities and supports both fine-tuning and RAG approaches depending on industrial use case requirements. Highlighting compelling customer success stories including Suncorp's employee productivity platform and AstraZeneca's transformation of 400,000 clinical trial documents into queryable insights, Craig revealed a striking reality about enterprise AI evaluation - that 85% of AI use cases are being evaluated only by the engineers who built them, reinforcing that proper evaluation frameworks remain foundational for trustworthy AI implementation. He concluded by introducing Agent Bricks as Databricks' evaluation-centric approach to building production agents, emphasizing that model flexibility and rigorous testing are essential for enterprises moving from experimentation to production, while sharing his vision that the industry must evolve from the "year of agents" to the "year of evaluation and quality."

Episode Highlights: [00:00] Quote of the Day by Craig Wiley [01:21] How Craig Wiley started his work in ML Ops that led him to Databricks [02:43] Data and AI layer connection creates order-of-magnitude acceleration [03:47] Mosaic AI acquisition expanded Gen AI solution capabilities [04:38] Classical ML statistics versus Gen AI evaluation challenges [05:48] Mosaic AI covers end-to-end from data ingestion [07:12] Training-serving skew eliminated through unified platform lineage [08:51] Fine tuning versus RAG depends on use case [10:49] Industrial agents benefit from fine-tuned smaller models [12:44] Common governance scheme covers tables through model access [13:52] Agent Bricks prioritizes accuracy over simplicity alone [15:44] Model flexibility crucial for speed and accuracy optimization [16:54] AB testing different models shows immediate performance differences [17:59] Suncorp and AstraZeneca demonstrate diverse AI applications [19:37] Asia Pacific shows aggressive AI adoption strategies [20:59] CFO approval requires proven agent effectiveness evaluation [22:00] 85% of AI cases evaluated only by building engineer [23:20] Model agnostic approach beats single-vendor AI strategies [24:12] Industry terminology evolves rapidly from RAG to agents [25:39] Customer creativity with governance capabilities inspires product development

Profile: Craig Wiley, Senior Director of Product Management at Databricks and Mosaic AI

LinkedIn: https://www.linkedin.com/in/craigwiley/

Podcast Information: Bernard Leong hosts and produces the show. The proper credits for the intro and end music are "Energetic Sports Drive." G. Thomas Craig mixed and edited the episode in both video and audio format.

Here are the links to watch or listen to our podcast.

Analyse Asia Main Site: https://analyse.asia

Analyse Asia Spotify: https://open.spotify.com/show/1kkRwzRZa4JCICr2vm0vGl

Analyse Asia Apple Podcasts: https://podcasts.apple.com/us/podcast/analyse-asia-with-bernard-leong/id914868245

Analyse Asia LinkedIn: https://www.linkedin.com/company/analyse-asia/

Analyse Asia X (formerly known as Twitter): https://twitter.com/analyseasia

Sign Up for Our This Week in Asia Newsletter: https://www.analyse.asia/#/portal/signup

Subscribe Newsletter on LinkedIn https://www.linkedin.com/build-relation/newsletter-follow?entityUrn=7149559878934540288