Evals: Ops QA and Multi-Turn Testing

From building internal AI labs to becoming CTO of Brex, James Reggio has helped lead one of the most disciplined AI transformations inside a real financial institution where compliance, auditability, and customer trust actually matter.

We sat down with Reggio to unpack Brex’s three-pillar AI strategy (corporate, operational, and product AI) [https://www.brex.com/journal/brex-ai-native-operations], how SOP-driven agents beat overengineered RL in ops, why Brex lets employees “build their own AI stack” instead of picking winners [https://www.conductorone.com/customers/brex/], and how a small, founder-heavy AI team is shipping production agents to 40,000+ companies. Reggio also goes deep on Brex’s multi-agent “network” architecture, evals for multi-turn systems, agentic coding’s second-order effects on codebase understanding, and why the future of finance software looks less like dashboards and more like executive assistants coordinating specialist agents behind the scenes.

We discuss:

• Brex’s three-pillar AI strategy: corporate AI for 10x employee workflows, operational AI for cost and compliance leverage, and product AI that lets customers justify Brex as part of their AI strategy to the board

• Why SOP-driven agents beat overengineered RL in finance ops, and how breaking work into auditable, repeatable steps unlocked faster automation in KYC, underwriting, fraud, and disputes

• Building an internal AI platform early: LLM gateways, prompt/version management, evals, cost observability, and why platform work quietly became the force multiplier behind everything else

• Multi-agent “networks” vs single-agent tools: why Brex’s EA-style assistant coordinates specialist agents (policy, travel, reimbursements) through multi-turn conversations instead of one-shot tool calls

• The audit agent pattern: separating detection, judgment, and follow-up into different agents to reduce false negatives without overwhelming finance teams

• Centralized AI teams without resentment: how Brex avoided “AI envy” by tying work to business impact and letting anyone transfer in if they cared deeply enough

• Letting employees build their own AI stack: ChatGPT vs Claude vs Gemini, Cursor vs Windsurf, and why Brex refuses to pick winners in fast-moving tool races

• Measuring adoption without vanity metrics: why “% of code written by AI” is the wrong KPI and what second-order effects (slop, drift, code ownership) actually matter

• Evals in the real world: regression tests from ops QA, LLM-as-judge for multi-turn agents, and why integration-style evals break faster than you expect

• Teaching AI fluency at scale: the user → advocate → builder → native framework, ops-led training, spot bonuses, and avoiding fear-based adoption

• Re-interviewing the entire engineering org: using agentic coding interviews internally to force hands-on skill upgrades without formal performance scoring

• Headcount in the age of agents: why Brex grew the business without growing engineering, and why AI amplifies bad architecture as fast as good decisions

• The future of finance software: why dashboards fade, assistants take over, and agent-to-agent collaboration becomes the real UI

—

James Reggio

• X: https://x.com/jamesreggio

• LinkedIn: https://www.linkedin.com/in/jamesreggio/

Where to find Latent Space

• X: https://x.com/latentspacepod

• Substack: https://www.latent.space/

Chapters

• 00:00:00 Introduction
• 00:01:24 From Mobile Engineer to CTO: The Founder's Path
• 00:03:00 Quitters Welcome: Building a Founder-Friendly Culture
• 00:05:13 The AI Team Structure: 10-Person Startup Within Brex
• 00:11:55 Building the Brex Agent Platform: Multi-Agent Networks
• 00:13:45 Tech Stack Decisions: TypeScript, Mastra, and MCP
• 00:24:32 Operational AI: Automating Underwriting, KYC, and Fraud
• 00:16:40 The Brex Assistant: Executive Assistant for Every Employee
• 00:40:26 Evaluation Strategy: From Simple SOPs to Multi-Turn Evals
• 00:37:11 Agentic Coding Adoption: Cursor, Windsurf, and the Engineering Interview
• 00:58:51 AI Fluency Levels: From User to Native
• 01:09:14 The Audit Agent Network: Finance Team Agents in Action
• 01:03:33 The Future of Engineering Headcount and AI Leverage