How Claude executes the drawing plan

Andrew describes Claude for Chrome planning, click-and-drag precision, and stepwise 'thinking out loud'.

Play episode from 16:04

chevron_right

Transcript

chevron_right

Transcript

Episode notes

In this episode of AI Tinkerers One-Shot, Joe sits down with Andrew Baker—serial builder, former Twilio engineer, and hands-on experimenter in agentic systems—to explore the rapidly evolving frontier of browser automation and AI-driven agents.

Andrew shares how his journey began with simple scripting experiments and gradually evolved into sophisticated browser agents capable of handling complex, real-world workflows. One standout example: an airline seat selector that used browser agents to secure optimal seats for frequent flyers—highlighting both the power and the limitations of today’s tooling.

Along the way, Andrew breaks down the practical challenges builders face when working with browser agents at scale:

• Vision model accuracy and UI interpretation

• DOM complexity and brittle page structures

• Authentication hurdles and session persistence

• The real economics of running large-scale automations

The conversation then shifts to “Claude Draws,” Andrew’s playful yet technically impressive side project that brings the classic 90s app Kid Pix into the age of AI. He explains how he wired up a remote PC, streamed sound output, and carefully crafted prompts that allow Anthropic’s browser agent to control a nostalgic art application—brushes, stamps, chaos, and all. The result is both a technical deep dive and a reminder that creativity is often where agentic tooling shines most.

Joe and Andrew also zoom out to examine the broader ecosystem shaping the future of browser-native agents. They discuss why UI accessibility matters for agents, how frameworks like Stagehand and Playwright are transforming automation workflows, and why personal evaluation benchmarks are becoming essential for builders pushing these systems beyond demos and into real usage.

💡 Resources & Links

Andrew Baker: https://www.linkedin.com/in/andrewtorkbaker

AI Tinkerers: https://aitinkerers.org

Andrew’s newsletter: https://implausible.ai

What you’ll learn

• How browser automation evolved from basic scripts to autonomous agents

• Why DOM parsing, vision models, and page structure still trip up agents

• How Claude for Chrome was used to control a web-based Kid Pix experience

• The architecture behind remote execution, sound streaming, and automation hacks

• How Stagehand and Playwright support modern browser automation

• The technical, economic, and ethical considerations shaping the future of browser agents

Chapters