Metrics, evals and product decision making

How Shopify Is Building The Future of AI Commerce with Andrew McNamaraShopify's Director of Applied ML, Andrew McNamara, reveals how the new Sidekick Pulse and SimGym features are revolutionizing e-commerce for merchants of all sizes. With 15 years of experience building AI assistants—dating back to pre-Siri days—Andrew breaks down how Shopify is using "LLM as a judge" to ensure quality and why "vibe entrepreneurship" is the future of business.Andrew explains how Sidekick has evolved from a simple chatbot into a proactive "AI Co-founder" that can democratize data for small business owners. He shares behind-the-scenes details on Sidekick Pulse, which performs deep research to surface actionable insights (like finding shipping errors that cost sales), and Simgym, a powerful simulator that uses AI shoppers to A/B test store changes before they go live.We also dive into the technical side of how Shopify evaluates these models using statistical rigor and why the ability for merchants to build admin apps with a single prompt is a game-changer for productivity.---Topics Covered- The 15-year evolution of AI assistants from rule-based systems to LLMs- How Sidekick Pulse proactively finds and fixes business-critical errors- Simgym: Using AI shoppers to simulate A/B tests without risking live traffic- The "LLM as a Judge" framework Shopify uses for product quality control- "Vibe Entrepreneurship" and reducing technical barriers for founders- Building custom Admin Apps in seconds using natural language prompts- Real-world examples from Andrew's own maple syrup store- The difference between general chatbots (Copilot) and specialized agents (Sidekick)- How "App Gen" allows merchants to create custom workflows instantlyEpisode Timestamps00:03 - Introduction to Andrew McNamara and his 15-year history with AI01:01 - Comparing early AI assistants (BlackBerry/Samsung) to modern LLMs06:45 - How Sidekick democratizes data analytics for small merchants11:25 - Deep Dive: Sidekick Pulse and proactive business research15:43 - Using "LLM as a Judge" to replace human evaluation at scale18:41 - Generating custom Admin Apps with a single prompt ("App Gen")21:58 - Andrew's personal experience running a Shopify store25:31 - The concept of "Vibe Entrepreneurship" as a North Star26:30 - Using AI to edit online store themes and layouts in real-time37:39 - Simgym: Simulating buyer behavior to predict experiment results42:06 - Why simulation is critical for both small and large enterprise merchants48:04 - The culture of technical depth and passion at Shopify51:34 - Why Andrew has dedicated his entire career to building assistants---## About Andrew McNamaraAndrew McNamara is the Director of Applied Machine Learning at Shopify, where he leads the development of Sidekick and other intelligent merchant features. Previously the Director of the Montreal Research Lab at Microsoft and a key contributor to Bing Chat (Copilot), Andrew has over 15 years of experience building and deploying AI assistants at scale.Shopify is the leading global commerce company, providing trusted tools to start, grow, market, and manage a retail business of any size. It powers millions of businesses in more than 175 countries and offers a unified platform for physical and digital commerce.Resources Mentioned- Shopify Sidekick (AI Commerce Assistant)- Sidekick Pulse (Proactive Research Agent)- Simgym (AI Shopper Simulator)- Microsoft Copilot & Bing Chat- "LLM as a Judge" Evaluation Framework---Partner Links- Book Enterprise Training — https://www.upscaile.com/- Subscribe to our free newsletter — https://www.theaireport.ai/subscribe-theaireport-youtube