Why Measuring Generative AI Is Hard

Is “AI-first” the future of business or just another tech buzzword?

On this episode of Digital Disruption, we’re joined by former Google Chief Decision Scientist and CEO of Kozyr, Cassie Kozyrkov.

Cassie is best known for founding the field of Decision Intelligence and serving as Google’s first Chief Decision Scientist, where she helped lead the company’s AI-first transformation. A sought-after advisor and keynote speaker, Cassie has guided organizations including Gucci, NASA, Meta, Spotify, Salesforce, and GSK on AI strategy. She combines deep technical expertise with theater-trained charisma to make complex concepts engaging and actionable for executive and general audiences alike delighting audiences in over 40 countries across all seven continents, including stages at the UN, WEF, Web Summit, and SXSW.

Cassie sits down with Geoff to unpack the hidden cost of the “AI-first” hype, the dangers of AI infrastructure debt, and why real AI readiness starts with people, not technology. She reveals how leaders can architect their organizations for innovation, build human-in-the-loop systems, and create cultures that embrace experimentation instead of fearing mistakes.

Cassie exposes why 95% of organizations fail to achieve measurable ROI from AI and how leaders can finally bridge the AI value gap. This conversation dives into why AI success isn’t about tools, it’s about leadership, measurement, and mindset.

Most organizations chasing “AI transformation” see no measurable ROI not because the technology fails, but because leaders are still measuring value the old way. Generative AI success is hard to quantify when there isn’t a single “right answer,” yet many businesses keep trying to apply outdated metrics to a completely new paradigm.

In this video:

00:00 Intro

00:44 The Generative AI Value Gap: Why 95% get no ROI

02:20 The paradox of AI productivity

05:38 Why measuring AI value is harder than we think

12:04 Leadership abdication: “Just sprinkle AI on everything”

15:10 AI infrastructure debt explained

20:17 What real AI readiness looks like (beyond tech)

23:42 Humans as part of AI infrastructure

28:00 Why “AI-first” isn’t one-size-fits-all

33:31 Building human judgment into AI systems

36:19 The risks of scaling too fast

41:34 Automation vs augmentation: where leaders go wrong

44:00 The “do the work” approach to AI success

48:35 The recipe for an AI-ready organization

53:40 Guardrails, governance, and security in AI systems

57:00 Thinking probabilistically: a new mindset for leaders

1:03:20 The human side of AI transformation

1:06:45 Leading through uncertainty

Connect with Cassie:

Website: https://www.kozyr.com/about

LinkedIn: https://www.linkedin.com/in/kozyrkov/

X: https://x.com/decisionleader

YouTube: https://www.youtube.com/c/Kozyrkov

Visit our website: https://www.infotech.com/?utm_source=youtube&utm_medium=social&utm_campaign=podcast

Follow us on YouTube: https://www.youtube.com/@InfoTechRG