How to Build ChatGPT Apps (The Next App Store?) | Live Demo by Colin Matthews
The Growth Podcast
Auto evals and fixing tool selection
Colin demos auto-evals where an LLM predicts the correct tool and highlights mismatches to improve descriptions.
Today’s Episode
ChatGPT Apps might be the next billion-dollar opportunity.
Or they might be another ChatGPT feature that gets abandoned in 6 months.
I genuinely don’t know yet.
But when people say “this could be the new App Store,” my ears perk up. I spent four years building an iOS app in the early days of the App Store. The distribution was incredible. We grew fast purely because of where we were.
So when OpenAI announced the ChatGPT App Store, I needed to understand it.
I brought in Colin Matthews to break it down. Colin is one of my go-to sources for technical product topics. Our AI prototyping collaborations have been some of your favorite episodes.
Today, we’re exploring ChatGPT Apps and what they mean for you as a product builder.
----
Check out the conversation on Apple, Spotify and YouTube.
Brought to you by:
* Maven: Get $500 off with my code on Coil Build ChatGPT Apps course
* Vanta: Automate compliance, Get $1,000 with my link
* Land PM job: 12-week experience to master getting a PM job
* Mobbin: Discover real-world design inspiration
* NayaOne: Airgapped cloud-agnostic sandbox
----
If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, and Mobbin - for free, grab Aakash’s bundle.
Are you searching for a PM job? Join me + 29 others for an intensive 12-week experience to master getting a PM job. Only 23 seats left.
----
Key Takeaways:
1. ChatGPT apps = MCP + widgets - The Model Context Protocol (invented by Anthropic) lets AI agents call external tools. OpenAI added UI widgets on top to create embedded app experiences directly in chat.
2. 900M weekly active users = massive distribution opportunity - This is the new SEO. Early data shows 26% higher conversion from AI traffic vs traditional search. Every enterprise will eventually build here.
3. You're building for multiple platforms - MCP works across ChatGPT, Claude (coming soon), Cursor, and other AI tools. Build once, distribute everywhere. Gemini doesn't support it yet.
4. Apps get called based on tool descriptions - Your metadata matters. Like SEO but for LLMs. Run evals to test if correct prompts trigger your tools. Iterate on descriptions to improve discovery.
5. Three eval categories: direct, indirect, negative - Direct: user names your app. Indirect: user describes outcome. Negative: irrelevant request shouldn't trigger your tool. Test all three systematically.
6. PMs should prototype but engineers ship production - Use tools like Chippy to prototype quickly and test concepts. Show stakeholders real interactions. Engineering team builds the production version.
7. Enterprise-first, solo builders second - Large companies (Target, Uber, Canva) are early adopters chasing distribution. But huge opportunity for indie builders once public marketplace launches.
8. Best opportunities: embedded collaboration tools - Spreadsheets, task lists, whiteboards where ChatGPT can partner with you. Not just search results—actual interactive experiences.
9. Error analysis on observability logs is critical - Track what prompts triggered which tools with what parameters. Look for mismatches between expected and actual behavior. Iterate tool descriptions.
10. Marketplace launching by end of 2024/early 2025 - Currently only launch partners can publish. Public marketplace coming soon means anyone can ship apps and reach ChatGPT's massive user base.
----
Where to Find Colin
Related Content
Newsletters:
* Complete Course: AI Product Management
Podcasts:
* Everything You Need to Know About AI
----
PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!
If you want to advertise, email productgrowthppp at gmail.
The AI-powered Podcast Player
