AI Agent Development Tradeoffs You NEED to Know

Sherwood Callaway, tech lead at 11X, joins us to talk about building digital workers—specifically Alice (an AI sales rep) and Julian (a voice agent)—that are shaking up sales outreach by automating complex, messy tasks.

He looks back on his YC days at OpKit, where he first got his hands dirty with voice AI, and compares the wild ride of building voice vs. text agents. We get into the use of Langgraph Cloud, integrating observability tools like Langsmith and Arize, and keeping hallucinations in check with regular Evals.

Sherwood and Demetrios wrap up with a look ahead: will today's sprawling AI agent stacks eventually simplify?

// Bio

Sherwood Callaway is an emerging leader in the world of AI startups and AI product development. He currently serves as the first engineering manager at 11x, a series B AI startup backed by Benchmark and Andreessen Horowitz, where he oversees technical work on "Alice", an AI sales rep that outperforms top human SDRs.

Alice is an advanced agentic AI working in production and at scale. Under Sherwood’s leadership, the system grew from initial prototype to handling over 1 million prospect interactions per month across 300+ customers, leveraging partnerships with OpenAI, Anthropic, and LangChain while maintaining consistent performance and reliability. Alice is now generating eight figures in ARR.

Sherwood joined 11x in 2024 through the acquisition of his YC-backed startup, Opkit, where he built and commercialized one of the first-ever AI phone calling solutions for a specific industry vertical (healthcare). Prior to Opkit, he was the second infrastructure engineer at Brex, where he designed, built, and scaled the production infrastructure that supported Brex’s application and engineering org through hypergrowth. He currently lives in San Francisco, CA.

// Related Links

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Sherwood on LinkedIn: /sherwoodcallaway/

#aiengineering

Timestamps:

[00:00] AI Takes Over Health Calls

[05:05] What Can Agents Really Do?

[08:25] Who’s in Charge—User or Agent?

[11:20] Why Graphs Matter in Agents

[15:03] How Complex Should Agents Be?

[18:33] The Hidden Cost of Model Upgrades

[21:57] Inside the LLM Agent Loop

[25:08] Turning Agents into APIs

[29:06] Scaling Agents Without Meltdowns

[30:04] The Monorepo Tangle, Explained

[34:01] Building Agents the Open Source Way

[38:49] What Production-Ready Agents Look Like

[41:23] AI That Fixes Code on Its Own

[43:26] Tracking Agent Behavior with OpenTelemetry

[46:43] Running Agents Locally with Phoenix

[52:55] LangGraph Meets Arise for Agent Control

[53:29] Hunting Hallucinations in Agent Traces

[56:45] Off-Script Insights Worth Hearing