The Real Python Podcast Configuring a Versatile LLM Harness & Scraping the Web With Scrapy
19 snips
Jul 24, 2026 
Ayan Pahwa, developer advocate at Zyte and builder of scraping tools and LLM harnesses. He discusses creating a compact Scrapy extension and scalable scraping with Zyte. He walks through his homelab self-hosting setup. He explains harness design, moving from prompt to loop engineering, and running local/edge models.
AI Snips
Chapters
Transcript
Episode notes
Small Scrapy Extension That Alerts On Success
- Ayan built a tiny 75-line Scrapy extension that plays a triumphant sound when a crawl finishes successfully and a sad tone on failure.
- The extension listens to Scrapy signals, checks scraped item counts, and gives audible feedback so you don't babysit long crawls.
Use Zyte API To Avoid Brittle Selectors
- Zyte provides both a Scrapy-maintained scraping framework and a separate Zyte API that can auto-extract structured data and reduce brittle CSS/XPath selectors.
- Use Zyte API as a fallback for blocked pages or when you want robust, scalable page extraction without custom selectors.
NoAlgoTube Bypasses YouTube Recommendations
- Ayan built NoAlgoTube to bypass YouTube recommendations by aggregating only channels he cares about and showing new uploads on a local Docker-hosted site.
- The site falls back to scraping when a channel's RSS fails and deploys via a CI/CD pipeline to his NUC home server.