There's something in TollBit's latest State of the Bots report that really jumped out at me. The major AI companies—notably Meta, OpenAI, and Google—have essentially given themselves permission to ignore robots.txt for their user agents. In the big picture of AI scraping it might seem like a detail, but it feels to me like a turning point from liberally interpreting the mechanisms for protecting content to brazenly flouting them.
I put this question directly to TollBit CEO Toshit Panigrahi when he joined me on this week’s Media Copilot podcast. A little background on TollBit: It’s a startup that specializes in helping publishers monetize their content when AI bots come calling. As everyone knows, the AI internet is rising fast—people are using chatbots to search the web, and those chatbots are giving them answers, not links. And if there was any doubt that this was a fad or not very impactful, Tollbit's most recent State of the Bots report for the first quarter 2025 puts that notion to rest. AI scraping is way up, and it's very quickly becoming comparable to scraping from search engines like Google or Bing.
That's one of several eye-opening, and in some cases worrisome, data points in the report, and Toshit and I dive deep into the most important ones. Like I mentioned, AI companies are now starting to openly ignore long-standing internet standards, effectively letting them crawl any "publicly available" content with impunity. What happens if this trend is left unchecked, and what can publishers do about it? We get into all the details, and their implications.
If you're stressed about what agents are going to mean for the media ecosystem, you definitely don't want to miss this one.
This episode covers:
Why AI bots are ignoring robots.txt and what it means for publishers right now
How the rise of agent-based scraping is quietly reshaping the open web
The surprising data behind how much AI scrapers take vs. how little they give back
What a “new value exchange” with AI looks like and how publishers might get paid
Where all this is headed if nothing changes — and what needs to happen next
🎙 Guest: Toshit Panigrahi
CEO of TollBit
Follow him on LinkedIn
And if you enjoyed this conversation, I’d encourage you to follow the show on Substack, Apple Podcasts, Spotify, or any other podcast app that you want. Also, I’d appreciate it if you’d leave a rating or review — it really does help the show. And if you’re on YouTube, don’t forget to “like” the video and subscribe to the channel 🔔
You can also subscribe to The Media Copilot newsletter (link in show notes) and visit mediacopilot.ai for exclusive resources, tools, and AI training courses built specifically for media professionals.
This episode of The Media CoPilot was produced by Pete Pachal, Executive Producer Michele Musso, and with video/audio editing by the Musso Media team. Produced by Musso Media. © 2025 Musso Media. All rights reserved.
Music: Favorite by Alexander Nakarada, licensed under Creative Commons by Attribution 4.0 License
© AnyWho Media 2025
