QA best practices and testing agents

Listen now: Spotify, Apple and YouTube

In the first-ever live recording of Supra Insider, Marc and Ben sat down with Jacob Bank, founder of Relay.app, to unpack one of the most urgent questions facing product leaders today: How do AI agents actually change the way we work? Instead of abstract predictions, Jacob shares the very real workflows, failures, and breakthroughs behind running a 10-person company that delegates work to more than 300 AI agents.

Across the conversation, the three dig into what PMs must learn next: writing job descriptions for agents, architecting responsibilities, managing automated execution, and understanding how agents influence velocity, product quality, and cross-functional collaboration. Jacob also discusses why PMs are lagging behind engineering and ops in adopting agentic workflows, and what will happen to teams who don’t catch up.

If you’re a PM, founder, or operator trying to understand how AI is reshaping product development, or you’ve struggled to translate “agent hype” into concrete, repeatable workflows, this episode gives you a realistic, practitioner-level framework for building with agents today, and preparing for what’s coming next.

All episodes of the podcast are also available on Spotify, Apple and YouTube.

New to the pod? Subscribe below to get the next episode in your inbox 👇

Interviewing at Uber, Figma, Stripe, or DoorDash?

If you’re a PM preparing for interviews at Uber, Figma, Stripe, and DoorDash… we think you’re going to love Insider Loops. You can read about it in Ben’s LI post, Marc’s LI post or check out the site directly below.

In this episode, we covered the following topics:

* (01:27) Jacob’s background and why PMs should care about AI agents

* (04:37) Why this session focuses on “hiring AI agents like employees”

* (07:38) The AI org chart explained — the Reddit marketing agent

* (15:05) Managing agent performance — firing, reviewing, and iterating

* (18:39) The hype cycle — why “AI agents are real” matters

* (25:55) What tasks should not be agents (one-offs, planning, magic tasks)

* (33:13) Building an AI agent live

* (37:42) Where agent outputs should land — Slack, Gmail, Docs

* (46:54) The six agents every PM should “hire” today

* (50:16) Deterministic vs agentic automations — the reliability spectrum

* (55:19) Making agents work in enterprise environments with locked-down systems

* (1:03:16) Live demo: Natural Language Builder using Relay.app

* Live Audience Q&A:

* (1:07:46) QA best practices — how to trust agents before delegating

* (1:09:21) Cost breakdown — why credits don’t matter if value compounds

* (1:12:16) The most expensive agent Jacob runs (customer health scoring

* (1:13:45) Why simple feedback loops fail — thumbs up/down isn’t enough

* (1:15:20) Agent drift, hallucination, precision, and the role of clarity

* And more!

Links:

* Relay.app: https://www.relay.app/

* Six PM Agents (slides Jacob showed in the session): https://ggl.link/six-pm-agents

* Jacob’s Maven Course: https://maven.com/jacob-bank/six-ai-agents-every-pm-needs/

* Jacob Bank: https://www.linkedin.com/in/jacobbank/

* Marc: https://www.linkedin.com/in/marcbaselga/

* Ben: https://www.linkedin.com/in/benerez/

If you’re a new subscriber, we encourage you to check out some of the recent episodes you might have missed:

To support the podcast, please check out the links below:

* Preparing for Meta-style Product Sense & Analytical Thinking interviews? Check out Ben’s 🤖 AI Practice Copilot for Meta-Style PM Interviews and self-paced course, Product Sense & Analytical Thinking Bootcamp for Meta-Style PM Interviews (25% off with code `supra insider` auto-applied here). Featured in Lenny’s Newsletter 🔥

* Supra has teamed up with Maven to bring you something special – courses that our own community members have personally curated. And because you’re part of the Supra family, you can get up to 35% in some of these handpicked selections with code SUPRAxMAVEN.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit suprainsider.substack.com