Technical challenges for browser agents

In this episode of AI Tinkerers One-Shot, Joe sits down with Andrew Baker—serial builder, former Twilio engineer, and hands-on experimenter in agentic systems—to explore the rapidly evolving frontier of browser automation and AI-driven agents.

Andrew shares how his journey began with simple scripting experiments and gradually evolved into sophisticated browser agents capable of handling complex, real-world workflows. One standout example: an airline seat selector that used browser agents to secure optimal seats for frequent flyers—highlighting both the power and the limitations of today’s tooling.

Along the way, Andrew breaks down the practical challenges builders face when working with browser agents at scale:

• Vision model accuracy and UI interpretation

• DOM complexity and brittle page structures

• Authentication hurdles and session persistence

• The real economics of running large-scale automations

The conversation then shifts to “Claude Draws,” Andrew’s playful yet technically impressive side project that brings the classic 90s app Kid Pix into the age of AI. He explains how he wired up a remote PC, streamed sound output, and carefully crafted prompts that allow Anthropic’s browser agent to control a nostalgic art application—brushes, stamps, chaos, and all. The result is both a technical deep dive and a reminder that creativity is often where agentic tooling shines most.

Joe and Andrew also zoom out to examine the broader ecosystem shaping the future of browser-native agents. They discuss why UI accessibility matters for agents, how frameworks like Stagehand and Playwright are transforming automation workflows, and why personal evaluation benchmarks are becoming essential for builders pushing these systems beyond demos and into real usage.

💡 Resources & Links

Andrew Baker: https://www.linkedin.com/in/andrewtorkbaker

AI Tinkerers: https://aitinkerers.org

Andrew’s newsletter: https://implausible.ai

What you’ll learn

• How browser automation evolved from basic scripts to autonomous agents

• Why DOM parsing, vision models, and page structure still trip up agents

• How Claude for Chrome was used to control a web-based Kid Pix experience

• The architecture behind remote execution, sound streaming, and automation hacks

• How Stagehand and Playwright support modern browser automation

• The technical, economic, and ethical considerations shaping the future of browser agents

Chapters

00:00:15 — Introduction and AI Tinkerers Community

02:49 — Twilio Origins and Browser Automation Journey

04:50 — Building the Airline Seat Selector

07:51 — Browser Agent Challenges and Vision Models

10:44 — Stagehand Framework and Browser Automation Stack

13:28 — Claude for Chrome and Authentication

16:58 — Kid Pix Origins and Demo Setup

21:33 — Technical Architecture and Playwright Tricks

29:24 — Evaluation Platform and Personal Benchmarks

37:42 — Future of Browser Agents and Web Economics

Subscribe for more conversations with the builders shaping the future of AI, automation, and agentic systems.