Marketplace Tech When an AI internet search competes against a human internet search
24 snips
May 1, 2025 Selena Deckelmann, Chief Product and Technology Officer at the Wikimedia Foundation, dives into the intriguing world of AI bots and their impact on Wikipedia. She discusses the unexpected traffic spikes following President Jimmy Carter's death, which exposed vulnerabilities in their system. Selena highlights the challenge of balancing public knowledge access with the ethical dilemmas posed by data scraping. Collaboration with developers is essential, as they navigate the complexities of AI and its growing influence on the internet.
AI Snips
Chapters
Transcript
Episode notes
Traffic Spike After Carter's Death
- When Jimmy Carter died, Wikipedia experienced a traffic spike that caused content bottlenecks and slow load times for about an hour.
- This event highlighted how sudden surges affect site performance despite Wikipedia's sophisticated caching and infrastructure.
AI Bots Increase Internet Scraping
- AI has massively increased web crawling to create vast training datasets for large language models.
- These bots scrape everything indiscriminately, unlike regular human users who focus on popular topics.
Wikimedia Requests Responsible Scraping
- Wikimedia asks companies scraping its content to collaborate, identify themselves, and follow licensing rules including attribution.
- Using Wikimedia Enterprise, a paid service, helps support the infrastructure and sustainability of Wikimedia projects.