Marketplace All-in-One When an AI internet search competes against a human internet search
May 1, 2025
Selena Deckelmann, Chief Product and Technology Officer at the Wikimedia Foundation, discusses the impact of AI bots on Wikipedia's infrastructure, particularly during spikes in traffic like after Jimmy Carter's passing. She highlights the challenges of commercial data scraping, emphasizing ethical concerns around AI training and the importance of equitable knowledge access. Deckelmann also addresses the balance between automated interactions and user experience, revealing the ongoing struggle to maintain site accessibility amid rising bot traffic.
AI Snips
Chapters
Transcript
Episode notes
AI Bots Overwhelm Wikipedia Servers
- The surge of AI bots scraping Wikipedia differs from regular human traffic spikes.
- Bots indiscriminately pull all data, increasing server loads and slowing access temporarily.
AI Training Drives Surging Web Crawling
- AI training drives massive, random web crawling to collect broad internet data.
- Wikipedia data, made by humans, is especially valuable for AI models to answer human questions.
Collaborate to Sustain Wikimedia Data
- Companies scraping Wikipedia should communicate and collaborate with Wikimedia Foundation.
- Applying best practices and attribution supports the sustainability of free knowledge.