AI Adoption Playbook Building AI agents that learn from feedback: BigPanda's drag-and-drop system | Alexander Page
The fastest path to production AI isn't perfect architecture, according to Alexander Page. It's customer validation. In his former role of Principal AI Architect at BigPanda, he transformed an LLM-based prototype into "Biggy," an AI system for critical incident management. BigPanda moved beyond basic semantic search to build agentic integrations with ServiceNow and Jira, creating AI that understands organizational context and learns from incident history while helping with the entire lifecycle from detection through post-incident documentation.
Alexander also gives Ravin BigPanda's framework for measuring AI agent performance when traditional accuracy metrics fall short: combine user feedback with visibility into agent decision-making, allowing operators to drag-and-drop incorrect tool calls or sequence errors. He reveals how they encode this feedback into vector databases that influence future agent behavior, creating systems that genuinely improve over time.
Topics discussed:
- LLM accessibility compared to traditional ML development barriers
- Fortune 500 IT incident management across 10-30 monitoring tools
- Building Biggy, an AI agent for incident analysis and resolution
- Customer-driven development methodology with real data prototyping
- Agentic integrations with ServiceNow and Jira for organizational context
- Moving beyond semantic search to structured system queries
- AI agent performance evaluation when accuracy is subjective
- User feedback mechanisms for correcting agent tool calls and sequences
- Encoding corrections into vector databases for behavior improvement
- Sensory data requirements for human-level AI reasoning
Listen to more episodes: