The fastest path to production AI isn't perfect architecture, according to Alexander Page. It's customer validation. In his former role of Principal AI Architect at BigPanda, he transformed an LLM-based prototype into "Biggy," an AI system for critical incident management. BigPanda moved beyond basic semantic search to build agentic integrations with ServiceNow and Jira, creating AI that understands organizational context and learns from incident history while helping with the entire lifecycle from detection through post-incident documentation.

Alexander also gives Ravin BigPanda's framework for measuring AI agent performance when traditional accuracy metrics fall short: combine user feedback with visibility into agent decision-making, allowing operators to drag-and-drop incorrect tool calls or sequence errors. He reveals how they encode this feedback into vector databases that influence future agent behavior, creating systems that genuinely improve over time.

Topics discussed:

LLM accessibility compared to traditional ML development barriers
Fortune 500 IT incident management across 10-30 monitoring tools
Building Biggy, an AI agent for incident analysis and resolution
Customer-driven development methodology with real data prototyping
Agentic integrations with ServiceNow and Jira for organizational context
Moving beyond semantic search to structured system queries
AI agent performance evaluation when accuracy is subjective
User feedback mechanisms for correcting agent tool calls and sequences
Encoding corrections into vector databases for behavior improvement
Sensory data requirements for human-level AI reasoning

Listen to more episodes:

Shadow Network: Media, Money, and the Secret Hub of the Radical Right with Anne Nelson