Battling AI Scrapers: Challenges and Innovations

We're joined by Xe Iaso, who discusses a creative solution to relentless AI bots and the unexpected delights of running an outrageously overpowered homelab.

Special Guest: Xe Iaso.

Sponsored By:

• Tailscale: Tailscale is a Zero config VPN. It installs on any device in minutes, manages firewall rules for you, and works from anywhere. Get 3 users and 100 devices for free.
• Unraid: A powerful, easy operating system for servers and storage. Maximize your hardware with unmatched flexibility.

Support Self-Hosted

Links:

• ⚡ Grab Sats with Strike Around the World — Strike is a lightning-powered app that lets you quickly and cheaply grab sats in over 110 countries.
• 🇨🇦 Bitcoin Well — Empowering sovereign ownership with free automatic self-custody 🔒
• 🎉 Boost with Fountain FM — Podcasts and music powered by community. Share, discover and support over 4 million shows and emerging artists.
• Xe Iaso's Blog
• Support Xe on Patreon
• Block AI scrapers with Anubis - Xe Iaso — I got tired of this and made a tool to stop them for good. I call it Anubis. Anubis weighs the soul of your connection using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots. It's a reverse proxy that requires browsers and bots to solve a proof-of-work challenge before they can access your site, just like Hashcash.
• Building native packages is complicated - Xe Iaso — Normally when I make projects, I don't expect them to take off. I especially don't expect to front page news on Ars Technica and TechCrunch within the span of a few days. I very much also do not expect to say sentences like "FFmpeg uses a program I made to help them stop scraper bots taking down their issue tracker". The last week has been fairly ridiculous in that regard.
• TecharoHQ/anubis — Anubis weighs the soul of your connection using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots.
• Open source devs are fighting AI crawlers with cleverness and vengeance — In a “cry for help” blog post in January, FOSS developer Xe Iaso described how AmazonBot relentlessly pounded on a Git server website to the point of causing DDoS outages. Git servers host FOSS projects so that anyone who wants can download the code or contribute to it.
• The surreal joy of having an overprovisioned homelab - Xe Iaso — Tonight I’ll tell you what gets me excited about my homelab and maybe inspire you to make your own. I'll get into what I like about it and clue you into some of the fun you get to have if one of your projects meant to protect your homelab goes hockey-stick.
• Talos Linux — Talos Linux is Linux designed for Kubernetes – secure, immutable, and minimal.
• Apollo: Sunshine fork — The easiest way to stream with the native resolution of your client device
• ansible-vault-bitwarden: Manage ansible-vault passwords securely with Bitwarden CLI
• Amcrest 4MP ProHD Indoor WiFi Camera — – 2. 4ghz WiFi IP camera features immaculate 4MP (2688x1520P at 30fps video using excellent low light capability utilizing the CMOS image sensor and chipset. Cover more ground using super-wide 90° viewing angle and remote pan/tilt.
• IPTV+: My Smart IPTV Player on the App Store
• Jupiter Broadcasting Meetup
• SSH 146 Boost Barn — Thank you to everyone who supported episode 146!