Season 6, Episode 16: Can an LLM evaluate ad creative? (with Luca Fiaschi)

Nov 4, 2025

Luca Fiaschi, a data science leader at PyMC Labs, dives into innovative ways machine learning can redefine ad evaluation. They discuss his recent paper on whether LLMs can replicate human purchase intent through scoring product concepts. Luca shares insights on how LLMs could streamline creative testing by using synthetic consumer models and Bayesian methods. The conversation highlights the challenges of creative volume and the promise of LLM-generated insights to enhance marketing strategies, making testing more efficient and cost-effective.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

ANECDOTE

Client Story: Colgate's Concept Overload

Luca describes Colgate sending thousands of AI-generated product concepts that overwhelmed their ability to run human panels.
PyMC Labs was asked to build synthetic consumers to prioritize which concepts go to expensive human testing.

ANECDOTE

PyMC Labs' Bayesian + AI Approach

PyMC Labs builds agents that wrap Bayesian models and synthetic consumers to provide interpretable decisions with uncertainty.
Luca joined PyMC Labs to bridge Bayesian modeling and generative AI for practical business decisions.

INSIGHT

Creative Volume Creates Testing Bottlenecks

Explosion of AI-created creative assets overwhelms existing testing and retrieval systems, raising testing costs.
Prioritizing what to create (not just what to test) becomes strategically important for advertisers.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

My guest on this episode of the podcast is Luca Fiaschi, a machine learning expert who previously held executive data science roles at MistPlay, StitchFix, and HelloFresh. Luca is now a Partner for the Generative AI vertical at PyMC Labs, a consultancy that specializes in the application of Bayesian methods to business problems and which maintains the open source PyMC library for Bayesian statistical modeling as well as the open source PyMC Marketing media mix modeling library.

The subject of my discussion with Luca is PyMC Labs' recent paper, LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings. I found the paper fascinating and wrote an overview on LinkedIn; the paper's authors (from PyMC Labs as well as Colgate) use an LLM to score product concepts, finding that the distribution of LLM-produced scores can be comparable to that produced by human panels.

Among other things, the podcast episode covers:

Background on the paper, and how the partnership with Colgate-Palmolive came about.
An overview of the LLM querying methodologies used in the paper and how they produced score distributions.
The approaches used in the paper for calculating similarities between the LLM-produced and human-produced scores.
The conclusions / findings of the paper.
What the implications of the paper are on marketing creative ideation and the use of LLMs for evaluating product and advertising concepts.

Thanks to the sponsors of this week’s episode of the Mobile Dev Memo podcast:

Xsolla. With the Xsolla Web Shop, you can create a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards, and analytics.
INCRMNTAL⁠⁠. True attribution measures incrementality, always on.
Universal Ads is Comcast’s self-serve TV ads platform that lets you launch campaigns in minutes across premium inventory from NBC, Paramount, Warner Bros. Discovery, Roku, and more.

Interested in sponsoring the Mobile Dev Memo podcast? Contact ⁠Marketecture⁠.