Evaluating AI: The Role of Product Management and Integrity

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

* Jira Product Discovery: Build the right thing, reliably

* AI PM Certification: Get $550 with code AAKASH550C7

* The AI Evals Course for PMs: Get $1050 off with code ag-product-growth

* Maven: Get $100 off my curation of their top courses

Today's Episode

Today’s guest is at the epicenter of AI - and he hasn’t done any podcasts before.

In today’s in-person chat, I sit down with Jake Brill, the Head of Integrity Product at OpenAI. He breaks down:

* The GPT-5 launch

* How OpenAI builds product

* What the PM role looks in the the future with AI

* The future of building product and the need to build agents into your product

* What it takes to break into OpenAI

If you've ever wondered what it takes to work at OpenAI or how to build AI products at scale, this episode is for you.

Your Newsletter Subscriber Bonus:

For subscribers, each episode I also write up a newsletter version of the podcast. Thank you for having me in your inbox.

(By the way, we’ve launched our podcast clips channel as well and we’re going to post most valuable podcast moments on this channel, so don’t miss out: subscribe here.)

1. Integrity’s Role in the GPT-5 Launch

Most PMs think about launching products in terms of features and marketing. But when you're serving hundreds of millions of users with breakthrough AI, the real challenge is infrastructure that can handle the surge without breaking.

OpenAI’s integrity team played 3 roles in GPT-5’s launch.

Pillar 1 - Identity Systems

Identity systems must scale from normal traffic to potentially 10x volume overnight. The technical challenge involves load balancing, database scaling, and ensuring your signup flow doesn't crash when everyone hits "Create Account" simultaneously.

Pillar 2 - Financial Infrastructure

Financial systems need bulletproof payment processing and fraud detection as conversions spike. This includes sophisticated fraud prevention - bad actors specifically target new model launches to exploit capabilities with stolen credit cards.

Pillar 3 - Safety Systems

Safety systems require multiple defense layers: model training, input/output classifiers, and behavioral monitoring. Red teaming happens during model training, at production checkpoints, and continuously post-launch.

What most PMs miss is that integrity isn't just defensive - it's an enabler of scale. Without rock-solid integrity infrastructure, even the most advanced AI models can't reach their intended audience.

Do you need an integrity team? If you're building consumer AI products at scale, handling sensitive data, or processing payments, the answer is probably yes.

2. How OpenAI Builds Product

OpenAI operates with a unique product philosophy that breaks traditional PM playbooks.

While most companies start with user problems and build solutions, OpenAI often starts with breakthrough capabilities and figures out how to bring them to humanity.

Let’s zoom in on 5 key takeways about how they build product:

Takeaway 1 - The Research-First Approach

Jake describes this inverted approach: "We've got the best researchers in the world building the most powerful AI capabilities in the world. And sometimes it's like, holy moly, we just had this big research breakthrough. How do we bring this capability to humanity?"

This research-first methodology requires unprecedented collaboration between product and research teams from day one, not as an afterthought.

Takeaway 2 - Planning That Embraces Uncertainty

Their planning process intentionally embraces uncertainty. Teams plan quarterly but assume only 60-70% completion rates.

"If you do anything more than that, it probably means you weren't being flexible enough to the needs of the business. If you do anything less than that, probably didn't do a great job forecasting."

Plans are written in pencil, not pen, with lightweight documents and async reviews wherever possible.

Takeaway 3 - Product Reviews Stay Startup-Style

Product reviews maintain startup-style directness despite OpenAI's scale. "People come in, it doesn't matter what level you are, you can talk directly with leadership. You don't have to have a fancy slide deck."

This creates trust through transparency and hiring excellence rather than process overhead.

Takeaway 4 - Heavy Slack Culture

OpenAI runs almost entirely on Slack. Jake estimates "conservatively like 90% of my written communication is in Slack."

They've built AI agents directly into their Slack channels for Q&A and operational tasks.

Takeaway 5 - Iterative Deployment Philosophy

The company's belief in iterative deployment shapes how they handle uncertainty. Rather than trying to predict every possible misuse case, they identify non-negotiable risks to mitigate before launch, build monitoring systems for edge cases, and "very quickly respond and build sophisticated solutions" based on real-world usage patterns.

"Actually, at the end of the day, it's really helpful to follow OpenAI's approach of iterative deployment, because once you start rolling things out, you can actually see in the real world how people accidentally might misuse your products."

3. What the PM Role Looks Like in the AI Future

Three major shifts define the future PM, Jake's perspective from the epicenter of AI development reveals:

Shift 1 - From Specification Writer to Evaluation Architect

The PM role is fundamentally shifting from specification writer to AI evaluation architect. Jake's team increasingly asks PMs to write evals because "they have the clearest vision of how the product should work in their head."

The evaluation writing skill becomes critical as AI products require objective measurement frameworks. This differs from traditional product metrics by focusing on capability assessment rather than just usage measurement.

Shift 2 - AI Prototyping Replaces Lengthy Specs

AI prototyping specifically transforms how PMs communicate vision.

Instead of lengthy written specifications, PMs can now build functional demonstrations. "Rather than just writing a proposal for how something works, just build a prototype of how something could work and you put that in people's hands."

This shift from description to demonstration accelerates feedback cycles and reduces interpretation gaps between teams.

Traditional PRDs still matter, but they're becoming AI-enhanced and less wordy. "I think they're gonna go hand in hand the prototypes and the PRDs. I do think PRDs will be less wordy because you won't have to spend as much time describing oh you click on this button and this thing happens you can just show people."

Shift 3 - The Human Elements Become More Important

But the human elements intensify rather than diminish. Jake emphasizes that empathy remains the most critical PM skill:

"Fundamentally, you are building products for people who are nothing like you. They may live in a different part of the world. They may be a different age, different gender."

In five years, Jake predicts PMs will need to manage not just humans, but agents.

4. The Future of Building Product and the Need to Build Agents Into Your Product

Every PM needs to think about building AI agents into their products - otherwise they’re missing out on the future of product.

Why Agents Matter

Jake frames this transition clearly:

"For those first couple of years, it's really been what we call assistance. You asked a model a question, you give it a prompt and you get a response…

But where we foresee this technology going is not just question and answer, but rather, here's a task. Can you please complete it for me?"

This transition requires rethinking product architecture at a fundamental level. Most digital products today are synchronous - "I take an action and a response or something else happens immediately."

Agent-first products embrace asynchronous complexity where "someone clicks button and something far more complex can happen behind the scenes and you don't have to sit there waiting for response."

Real-World Agent Implementation

Jake already demonstrates this shift in his daily work. He uses agents for recruiting ("here's sort of the properties. Ideally, they have X years of design experience... Please go help and source some candidates"), medical research, and market analysis.

For PMs specifically, agents excel at competitive analysis, presentation creation, and prototyping - areas where the combination of research depth and creative output provides immediate value.

The strategic imperative is clear: "If you're not thinking about how to build products that are agentic in their fundamental nature, you're probably A, not maximizing the power of this technology and B, you're probably building a product that's going to be obsolete in a shorter time horizon."

The Infrastructure Challenge

The challenge extends beyond individual products to ecosystem integration. As Jake notes, "there's not going to be just one company building agentic products" and "the failure state would be if there's not a standard language for all of them to talk together."

Products need to consider how their agents will communicate with other agents, requiring standards like MCP (Model Context Protocol) for tool integration and future protocols for agent-to-agent communication.

Companies building agent-first products must also prepare for agent reliability challenges. Jake discusses the emerging problem of deceptive behavior: "agents learning to cheat" requires multiple defense layers including alignment training, behavioral monitoring, and constant red teaming.

The solution involves "model training, model level classifiers, actor level classifiers, production monitoring, and then just like constant red teaming."

5. What It Takes to Break Into OpenAI

When I asked my newsletter subscribers for their dream company, OpenAI was the overwhelming #1 dream company 600 first-place votes compared to 200 for second place.

Jake's advice for getting into OpenAI?

* Start Building with AI: You really need to have a facility with AI products and models. There's no excuse for not—it's so easy to use these products, to vibe code, to play around with APIs.

* Get a Referral: When we post jobs publicly, a lot of people apply. It really helps if someone at the company has worked with you and can speak to your skill.

* Show Relevant Experience: Make your AI experience and domain expertise explicitly clear on your resume and LinkedIn.

* Broad Background Wins: The best candidates aren't just specialists. They have experience across different areas (fraud AND ads AND growth) giving them multiple perspectives on ambiguous problems.

Jake's own story is proof: he spent 2 months preparing while on paternity leave, treating the interview process with the intensity it deserved because "it felt like what I'd been working towards my whole career."

The full conversation covers Jake's journey from Facebook's first internal PM hire to surviving a traumatic brain injury to leading integrity at the world's most important AI company. Essential listening if you're building your career in AI products. Watch or listen now.

Key Takeaways

Where to find Jake

* LinkedIn

Related Content

Podcasts:

* Complete Course: AI Product Management

* 5 AI Agents Every PM Should Build

* Full Roadmap: Become an AI PM

Newsletters:

* How to become an AI Product Manager

* AI Agents: The Ultimate Guide for PMs

* The AI PM's Playbook: How Top PMs Are 10x-ing Their Impact

P.S. More than 85% of you aren't subscribed yet. If you can subscribe on YouTube, follow on Apple & Spotify, my commitment to you is that we'll continue making this content better.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe