He Built an AI Audience Simulator. It’s the Future of Customer Research. - Ep. 49 with Michael Taylor

10 snips

Feb 26, 2025

Michael Taylor, creator of Rally and co-author of a prompt engineering book, shares how he uses AI to replicate customer responses, revolutionizing market research. He discusses simulating audience behavior to A/B test headlines, ensuring valuable insights before launch. The conversation highlights AI's potential in understanding consumer profiles while stressing the importance of precise prompts. Michael's tinkerer mindset illuminates the challenges and future of AI, including balancing automation with human creativity, as he demonstrates the power of his innovative tool live.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

00:00 / 00:00

AI as Audience Simulator

AI can roleplay multiple characters to simulate a whole focus group or audience.
This allows testing assumptions risk-free before launching ideas to the real world.

00:00 / 00:00

AI Profile Beats Friends' Guesses

Dan Shipper used GPT-4 to create a personality profile from his tweets, then took a personality test pretending to be him.
The AI's responses overlapped more with his own than his girlfriend's and mom's attempts did.

00:00 / 00:00

Use Binary Judging Criteria

Break down judging complex tasks into many simple binary yes/no criteria for AI evaluation.
Aggregating many small rating scores yields more reliable and consistent results than broad numeric scales.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Michael Taylor has perfected the art of getting AI to speak in tongues. He’s taught it to mimic the voices of your customers—so you can see how they would respond before you ship.

Michael is the creator of Rally, a market research tool that lets you simulate an audience of AI personas. He built a simulator that lets us A/B test Every’s headlines on an audience that mimics the real Hacker News audience. It’s become a part of my writing workflow, and I love it because you test your assumptions quickly, cheaply, and without any of the risks of putting something out into the world.

Besides Rally, Michael co-authored a book on prompt engineering for O’Reilly, and he writes a column for Every about managing AI tools like you would people. In a past life, he founded a growth marketing agency which he grew to 50 people and sold in 2020. One of the reasons I’m drawn to Michael’s work is because he has a tinkerer’s mindset. He’s always exploring the limits of what a new technology can do, and what he’s into today, everyone else will likely discover six months later. We spent an hour talking about using language models to judge your work, best practices for assessing an AI’s performance, and Michael’s flow inside Cursor. He also demos Rally live on the show, testing three different potential headlines for an Every article.

If you found this episode interesting, please like, subscribe, comment, and share!

Want even more?

Sign up for Every to unlock our ultimate guide to prompting ChatGPT here: https://every.ck.page/ultimate-guide-to-prompting-chatgpt. It’s usually only for paying subscribers, but you can get it here for free.

To hear more from Dan Shipper:

Subscribe to Every: https://every.to/subscribe
Follow him on X: https://twitter.com/danshipper

Timestamps:

Introduction: 00:01:32
AI can simulate human personalities with remarkable precision: 00:04:30
How Michael simulated a Hacker News audience: 00:08:15
Push AI to be a good judge of your work: 00:15:04
Best practices to run evals: 00:19:00
How AI compresses years of learning into shorter feedback loops: 00:23:01
Why prompt engineering is becoming increasingly important: 00:27:01
Adopting a new technology is about risk appetite: 00:44:49
Michael demos Rally, his market research tool: 00:47:10
The AI tools Michael uses to ship new features: 00:54:53

Links to resources mentioned in the episode:

Michael Taylor: @hammer_mt
Join the waitlist for Rally, Michael’s synthetic market research tool: https://askrally.com/
The book Michael co-authored on prompt engineering: Prompt Engineering for Generative AI
The column Michael writes for Every: Also True for Humans
Michael’s article on personas of thought: "I Asked 100 AI Agents to Judge an Advertisement”
Michael’s article on building a Hacker News simulator: "I Created a Hacker News Simulator to Reverse-engineer Virality”