Agent Benchmarks Using Games and Play

The Good Stuff, with Pete and Andy - Episode 27: Lessons Learned

Hosts: Pete and Andy

Episode Overview: Pete and Andy reflect on lessons learned from months of experimentation with AI agents, coding tools, and building software. They discuss the shift from "vibe coding" to more structured approaches, the importance of shipping usable tools, and their plans for a multiplayer version of Wingman.

Key Discussion Points:

Side Quests and Experimentation (01:16-08:30)

• Andy builds a habit tracker that evolved into a doom-scrolling prevention app
• Pete experiments with media over QUIC and real-time streaming protocols
• The power of AI to remove gatekeeping from learning new technologies
• Building an Nostr-based virtual pub with spatial audio

Vibe Coding vs. Slow Coding (09:38-16:13)

• Insights from working with serious engineers on AI-assisted development
• Everyone uses AI differently - no single "vibe coding" workflow
• The importance of understanding your codebase architecture
• Moving slower to go faster: maintaining intuition while delegating implementation

Build vs. Buy: The Shopify Question (14:25-25:06)

• Andy's journey building an e-commerce site from scratch instead of using Shopify
• The value of understanding how things work vs. convenience of platforms
• Localization of software development - kids will build these things natively
• Self-reliance as a valuable use of AI-gifted time

Shipping Tools People Can Use (25:06-33:00)

• The critical lesson: put working demos in users' hands
• Plans for multiplayer Wingman to lower barriers to experimentation
• Designing the business into Wingman - mapping workflows and agents
• Testing at Bush Bash with live coding sessions

Orchestrators vs. Deterministic Processes (33:00-38:52)

• Why probabilistic orchestrator agents often fail in production
• The case for simple, deterministic workflow rules
• Left curve vs. mid curve: sometimes simpler is better
• Humans should still design the business processes

Rate Limits and Model Selection (38:52-42:43)

• Claude Haiku as a solution to usage limits
• Running agents via API for unlimited usage
• Multiplayer mode for sharing subscriptions efficiently
• The challenge of making complex technology accessible

Simplifying the Message (42:43-48:31)

• Beacon demo: focus on the "moment of magic" not the complexity
• "Solvatur Ambulando" - solve it by walking around
• Wingman's unique value: anywhere access + multiplayer agents
• Don't let ego get in the way of clear communication

AI Agents Playing Games (48:31-59:09)

• Using game environments to test model performance for business applications
• Games as sandboxes for learning resource allocation and strategic thinking
• Beyond single-agent approaches: teams of specialized agents
• General Catalyst's investment in gaming arenas for model testing

Multiple Minds Per Task (55:13-01:01:38)

• Humans have multiple personalities for different contexts
• Agents may need similar specialization to avoid being overwhelmed
• File-based handoffs between agents as a clean interface
• The power of forcing agents to document their reasoning

"Mid curve me is just like 'oh I've been so clever' - but that's not for the person on the other end that wants to look at it."