From Vibe Coding to Vibe Researching: OpenAI’s Mark Chen and Jakub Pachocki
The a16z Show
Evaluations: Beyond Saturated Benchmarks
Jakub and Mark discuss eval saturation, limits of standard benchmarks, and the need for economically meaningful evaluations.
What comes after vibe coding? Maybe vibe researching.
OpenAI’s Chief Scientist, Jakub Pachocki, and Chief Research Officer, Mark Chen, join a16z general partners Anjney Midha and Sarah Wang to go deep on GPT-5—how they fused fast replies with long-horizon reasoning, how they measure progress once benchmarks saturate, and why reinforcement learning keeps surprising skeptics.
They explore agentic systems (and their stability tradeoffs), coding models that change how software gets made, and the bigger bet: an automated researcher that can generate new ideas with real economic impact. Plus: how they prioritize compute, hire “cave-dweller” talent, protect fundamental research inside a product company, and keep pace without chasing every shiny demo.
Timecodes:
0:00 Introduction & Goals of Automated Researcher
0:43 The Evolution of Reasoning in AI
1:46 Evaluations: From Benchmarks to Real-World Impact
5:15 Surprising Capabilities of GPT-5
6:56 The Research Roadmap: Next 1, 2, 5 Years
7:46 Long-Horizon Agency & Model Memory
9:44 Reasoning in Open-Ended Domains
11:18 The Role and Progress of Reinforcement Learning
13:14 Reward Modeling & Best Practices
14:21 The New Codex: Real-World Coding
16:20 AI vs. Human Coding: The New Default
20:07 What Makes a Great Researcher?
21:14 Persistence, Conviction, and Problem Selection
26:00 Building and Sustaining a Winning Research Culture
31:45 Balancing Product and Fundamental Research
39:00 The Importance of Compute and Physical Constraints
45:50 Maintaining Speed and Learning at Scale
47:18 Trust and Collaboration at OpenAI
Resources:
Find Jakub on X: https://x.com/merettm
Find Mark on X: https://x.com/markchen90
Find Sarah on X: https://x.com/sarahdingwang
Find Anjney on X: https://x.com/AnjneyMidha
Stay Updated:
If you enjoyed this episode, be sure to like, subscribe, and share with your friends!
Find a16z on X: https://x.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z
Listen to the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX
Listen to the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711
Follow our host: https://x.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Stay Updated:
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Show on Spotify
Listen to the a16z Show on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
The AI-powered Podcast Player
