Inside Browser Automation: Andrew Baker on Agents, Playwright, and Claude Draws

Jan 16, 2026

Andrew Baker, a former Twilio engineer and creator of innovative projects like an airline seat selector, shares his fascinating journey into browser automation. He discusses the evolution of sophisticated agents that tackle real-world tasks and the technical hurdles they encounter, such as DOM complexity and authentication issues. Baker also showcases 'Claude Draws', his AI-enhanced revival of the classic Kid Pix app, revealing how nostalgia meets creativity through automation. Tune in for insights on shaping the future of browser-native agents!

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Airline Seat Snipe Success Story

Andrew built an airline seat selector that monitored seat maps and updated assignments when better seats opened.
He sold that product after proving it worked for frequent flyers and execs who valued better seats.

INSIGHT

Vision Limits Make Web Agents Brittle

Browser agents struggle because they rely heavily on vision models which are still limited and misalign with required precise interactions.
Parsing large DOMs hits token and context limits, making a purely LLM-driven approach brittle and costly.

ADVICE

Distill Repeated Flows Into Deterministic Code

Don't put an LLM in the driver's seat for every step; convert frequent workflows into deterministic browser automation.
Build tooling to make maintaining per-site automation practical and cheaper than repeated vision calls.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this episode of AI Tinkerers One-Shot, Joe sits down with Andrew Baker—serial builder, former Twilio engineer, and hands-on experimenter in agentic systems—to explore the rapidly evolving frontier of browser automation and AI-driven agents.

Andrew shares how his journey began with simple scripting experiments and gradually evolved into sophisticated browser agents capable of handling complex, real-world workflows. One standout example: an airline seat selector that used browser agents to secure optimal seats for frequent flyers—highlighting both the power and the limitations of today’s tooling.

Along the way, Andrew breaks down the practical challenges builders face when working with browser agents at scale:

• Vision model accuracy and UI interpretation

• DOM complexity and brittle page structures

• Authentication hurdles and session persistence

• The real economics of running large-scale automations

The conversation then shifts to “Claude Draws,” Andrew’s playful yet technically impressive side project that brings the classic 90s app Kid Pix into the age of AI. He explains how he wired up a remote PC, streamed sound output, and carefully crafted prompts that allow Anthropic’s browser agent to control a nostalgic art application—brushes, stamps, chaos, and all. The result is both a technical deep dive and a reminder that creativity is often where agentic tooling shines most.

Joe and Andrew also zoom out to examine the broader ecosystem shaping the future of browser-native agents. They discuss why UI accessibility matters for agents, how frameworks like Stagehand and Playwright are transforming automation workflows, and why personal evaluation benchmarks are becoming essential for builders pushing these systems beyond demos and into real usage.

💡 Resources & Links

Andrew Baker: https://www.linkedin.com/in/andrewtorkbaker

AI Tinkerers: https://aitinkerers.org

Andrew’s newsletter: https://implausible.ai

What you’ll learn

• How browser automation evolved from basic scripts to autonomous agents

• Why DOM parsing, vision models, and page structure still trip up agents

• How Claude for Chrome was used to control a web-based Kid Pix experience

• The architecture behind remote execution, sound streaming, and automation hacks

• How Stagehand and Playwright support modern browser automation

• The technical, economic, and ethical considerations shaping the future of browser agents

Chapters

00:00:15 — Introduction and AI Tinkerers Community

02:49 — Twilio Origins and Browser Automation Journey

04:50 — Building the Airline Seat Selector

07:51 — Browser Agent Challenges and Vision Models

10:44 — Stagehand Framework and Browser Automation Stack

13:28 — Claude for Chrome and Authentication

16:58 — Kid Pix Origins and Demo Setup

21:33 — Technical Architecture and Playwright Tricks

29:24 — Evaluation Platform and Personal Benchmarks

37:42 — Future of Browser Agents and Web Economics

Subscribe for more conversations with the builders shaping the future of AI, automation, and agentic systems.