AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
2min chapter
Talking Drupal #453 - Urban Institute
Talking Drupal
Challenges with Performance and AI Bots affecting Editing Process
Exploring editors' challenges with editing and saving consistently due to Cloudbot generating 600 requests per minute, amidst issues with Cloudflare's global rules impacting the editing process and efforts to reduce strain by limiting AI bot requests.
00:00
Transcript
Episode notes
Speaker 3
Well, just, just jumping off what you said, Josh, because you wanted to jump into that a little bit. What's your, what's your number one pain point that you have right now for your editors that you wish you could just solve? Maybe our listeners have a module like access unpublished that you just hadn't heard of the number one's an issue for you.
Speaker 1
The number one at the moment, and oh my, is if I could wave my wand and it would just work, is performance. And that actually brings us back to the thing I kind of talked a little bit about, which is from Anthropic AI, has been hitting urban .org, which is an elite, it's actually higher than elite, there's like multiple tiers of elite Pantheon. It's an elite Pantheon site, it stays up even when Cloudbot's hitting us with 600 requests a minute. But when when Cloudbot's hitting us with 600 requests a minute, normally we about 100 requests a minute. So about five times our normal traffic sustained over like 72 hours. And when, when that's happening, our editors cannot click edit and save consistently, they get, they get all sorts of errors and what happened, the reason that is, is we're using Cloudflare. has a global rules about like how long a process can run before it gives up and it'll cut you off on the front end but the back end. Is still saving or whatever it might have been successful if you could wait for ten seconds but cuts you off with whatever the global value is, I think is the real problem. But really, honestly, Cloudbot, what are you doing? You're killing us. And so what we've implemented is a, we're looking for Cloudbot in the thing, because I don't want to, I don't want to put robots .txt in there. We're not, we're not saying AI can't have access to all of our research. We want to be part of the conversation. But we are creating rules that AI bots can only have access. I think it's like 10 requests a minute or something like that. We're taking them from the 600 that they want to 10. If
Speaker 4
they can't use more than 50% of your capacity. I
Speaker 1
would love to be able to write a rule like that. That would be great.
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode