No Hacks: Optimising the Web for AI Agents 215: The Agent-Broken Web - Why AI Can't See Your Website
Jan 28, 2026
29:40
Your website might rank #1 on Google but be completely invisible to ChatGPT, Claude, and Perplexity. In this episode, let's break down why a huge chunk of the web is fundamentally broken for AI systems - not because of bad content, but because of technical decisions that made sense for humans but make sites invisible to the AI systems rapidly becoming the front door to the internet.
Chapter Timestamps
- 00:00:00 - Introduction: The new game your website is losing
- 00:01:43 - The Scale of the Problem: AI crawler traffic explosion
- 00:05:19 - The JavaScript Problem: Why AI crawlers can't see your content
- 00:10:28 - The Bot Protection Paradox: Accidentally blocking AI
- 00:14:40 - The Speed Requirement: Why 200ms matters
- 00:17:46 - AI Agents Are Struggling Too: Browser agents and their limitations
- 00:20:46 - How to Fix It: 6 things you need to do
- 00:25:33 - Closing: The web is adapting again
Key Statistics
- 569 million GPTBot requests on Vercel's network in a single month
- 370 million ClaudeBot requests in the same period
- 305% growth in GPTBot traffic (May 2024 to May 2025)
- 157,000% increase in PerplexityBot requests year-over-year
- 33% of organic search activity now comes from AI agents
- ~40% failure rate for the best AI browser agents on complex tasks
The 6 Things to Fix
- Implement Server-Side Rendering (SSR) - If your site uses a JavaScript framework (React, Vue, Angular) with client-side rendering, switch to SSR or static site generation immediately. Use Next.js, Nuxt, or a pre-rendering service.
- Add Structured Data with JSON-LD - Expose key information in machine-readable format using schema.org markup. Microsoft confirmed Bing uses this to help Copilot understand content.
- Optimize for Speed - Target server response time under 200ms. First Contentful Paint under 1 second. Largest Contentful Paint under 2.5 seconds.
- Check Your Bot Protection Settings - Review Cloudflare, AWS WAF, or your CDN's bot management. Make a deliberate decision about GPTBot, ClaudeBot, and PerplexityBot access.
- Kill Infinite Scroll and Lazy Loading for Content - Use paginated URLs with standard HTML links. Ensure high-value content is in the initial HTML response.
- Keep Sitemaps Current - Maintain proper redirects, consistent URL patterns, and fix broken links.
Tools Mentioned
- Glimpse - Free tool to test how AI sees your website: glimpse.webperformancetools.com
Show Links
Sources Referenced in This Episode
AI Crawler Statistics:
- Vercel Blog - The Rise of the AI Crawler
- Cloudflare 2025 Year in Review
- Cloudflare - From Googlebot to GPTBot
- Search Engine Land - AI Optimization Guide
JavaScript Rendering:
No Hacks is a podcast about web performance, technical SEO, and the agentic web. Hosted by Slobodan "Sani" Manic.