AI + a16z

Benchmarking AI Agents on Full-Stack Coding

Mar 28, 2025

Sujay Jayakar, co-founder and Chief Scientist at Convex, dives into the future of autonomous coding. He discusses the challenges AI agents face with full-stack development and the significance of robust evaluation methods like Fullstack Bench. Jayakar emphasizes how type safety can reduce errors and improve consistency. He shares insights on which AI models excel in real-world app-building, and why treating your toolchain as part of the prompt could transform development workflows. Perfect for developers looking to enhance their projects with AI!

33:28

Creator website

Episode guests

Sujay Jayakar

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Effective trajectory management in AI coding is essential for decision-making and navigating complex coding paths towards successful application development.

Implementing type safety and robust evaluation processes significantly enhances AI coding agents' reliability and performance in real-world app-building tasks.

Deep dives

The Complexity of Trajectory Management in AI Coding

Trajectory management remains an underdeveloped field in AI coding, resembling the challenge of navigating from a starting position to an end goal with often unclear pathways. Establishing effective heuristics for coding is complex, as it involves understanding when to commit to a coding path to ensure successful progression. Traditional coding education emphasizes impulse control and decision-making in committing to designs, mirroring game strategies. This complexity highlights the need for improved trajectory management that can assist AI agents in making more informed coding decisions.

Intro

2min

Revolutionizing Application Development with Reactive Databases

2min

Enhancing AI in Full-Stack Development

15min

Benchmarking AI Models

4min

Benchmarking AI in Full-Stack Coding

8min

Maximizing Coding Efficiency with AI Tools

2min

In this episode, a16z General Partner Martin Casado sits down with Sujay Jayakar, co-founder and Chief Scientist at Convex, to talk about his team’s latest work benchmarking AI agents on full-stack coding tasks. From designing Fullstack Bench to the quirks of agent behavior, the two dig into what’s actually hard about autonomous software development, and why robust evals—and guardrails like type safety—matter more than ever. They also get tactical: which models perform best for real-world app building? How should developers think about trajectory management and variance across runs? And what changes when you treat your toolchain like part of the prompt? Whether you're a hobbyist developer or building the next generation of AI-powered devtools, Sujay’s systems-level insights are not to be missed.

Drawing from Sujay’s work developing the Fullstack-Bench, they cover:

Why full-stack coding is still a frontier task for autonomous agents
How type safety and other “guardrails” can significantly reduce variance and failure
What makes a good eval—and why evals might matter more than clever prompts
How different models perform on real-world app-building tasks (and what to watch out for)
Why your toolchain might be the most underrated part of the prompt
And what all of this means for devs—from hobbyists to infra teams building with AI in the loop

Learn More:

Introducing Fullstack-Bench

Follow everyone on X:

Sujay Jayakar

Martin Casado

Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.