Bots are taking over the internet, previewing a new world where agents interact and humans consume bot-created content. Cloudflare, a leading content delivery network, saw this coming.
Cloudflare Chief Strategy Officer Stephanie Cohen told me that publishers reported less traffic to their sites and less human traffic. “Those numbers were 10 times worse than a decade ago.”
This has broken the open web trade of the crawling right for traffic promise. Cohen said, “For some bots, the crawl-to-traffic ratio is tens of thousands of times worse than it used to be.”
Five takeaways from our conversation:
- Scarcity is required. Cloudflare’s pay-per-crawl model aims to restore balance by creating enforceable scarcity. “If you don’t have scarcity, there’s no market. Changing the defaults was the only way for a market to develop.”
- AI engines need publishers. Models rely on fresh, high-quality publisher content to stay accurate. “If you got asked a question and didn’t know what happened over the last two weeks, your answer is going to be stupider.”
- Pay-per-crawl is a first step, not the finish line. Cloudflare’s system lets publishers decide whether to block, allow, or charge AI crawlers. It’s a way to test how a content economy might form. As Cohen put it, “It’s the simplest thing you can do because you can count a crawl.”
- The goal is the Spotify model. Cohen pointed to Spotify’s emergence after Napster as a precedent for compromise. “There’s lots of money going to creators in that model… it took a while for that market to develop.”
- The problem is Google. Publishers face a lose-lose choice: block Google entirely or allow AI Overviews to rewrite their work. “If they want search, they’re also in AI overviews… They can’t both crawl and pay nothing.”
