Tech Life Web-scraping bots
7 snips
Oct 28, 2025 Zoe West, a Senior researcher at Cornell, talks about AI's troubling effects on the fashion industry, revealing how AI models threaten real diversity and job opportunities. Alex Kekesi, from Pornhub, discusses challenges surrounding age verification for adult content, criticizing current methods for privacy issues. Selina Deckelman, from the Wikimedia Foundation, explains the significant impact of bot traffic on Wikipedia, leading to a decline in human visitation and stressing the need for better automated access solutions. It's a fascinating dive into the intersection of technology and ethics.
AI Snips
Chapters
Transcript
Episode notes
Wikipedia Traffic Drop From Automation
- Wikipedia has seen an ~8% decline in human traffic since May, driven in part by automated bots scraping content for AI use.
- Wikimedia changed bot-detection and created an Enterprise channel to separate robot access and protect human readers and infrastructure.
Bots Strain Wikipedia Infrastructure
- Bots scrape pages rapidly and uniformly, consuming disproportionate computing resources compared with human readers.
- Wikimedia says separating robot and human use cases reduces infrastructure cost and improves experience for people.
Provide A Dedicated Access Channel
- Use dedicated, managed access channels for heavy automated consumers to protect public services.
- Wikimedia's Enterprise offering redirects automated traffic to avoid harming human access and infrastructure.