The Stack Overflow Podcast Search engine bots crawled so AI bots could run
9 snips
Jan 6, 2026 Robert Lester, an Akamai data scientist and AI Pulse blogger, joins the discussion on the rapidly evolving landscape of AI bots. He delves into how these bots differ from traditional crawlers and reveals that AI bot traffic, while still a small percentage, has surged by 400%. Lester emphasizes the importance of understanding bot intent over mere classification, highlights the nuances of bot mitigation based on business models, and identifies commerce as a key target sector, showcasing how emerging agents are reshaping user interactions.
AI Snips
Chapters
Transcript
Episode notes
Classify Bots By Intent Not Just 'Bot'
- AI traffic includes distinct classes: traditional search crawlers, research/training bots, and user-driven fetchers that models invoke at runtime.
- Robert Lester says categorizing by intent and behavior matters more than a simple bot-or-not distinction.
Rapid Growth Outpaces Current Volume
- AI bot traffic is still a small percent of validated bot traffic but has grown rapidly year-over-year.
- Akamai measured about a 400% increase across industries from the prior year, so growth rate is the key concern.
Mitigate Bots Based On Business Goals
- Choose mitigation posture based on your business model and whether content drives referrals or sales.
- Robert Lester recommends nuanced, case-by-case management rather than blanket blocking.