Syntax - Tasty Web Development Treats

763: Web Scraping + Reverse Engineering APIs

44 snips
May 1, 2024
Explore the fascinating world of web scraping, from setting up tools to navigating protected routes. Learn effective techniques for data extraction and the importance of understanding APIs. Discover how to locate web elements, even when classes are missing, using XPath and ARIA labels. Delve into the ethical challenges of automation, like circumventing CAPTCHAs, and hear humorous anecdotes linking family life to tech struggles. Plus, check out tips for optimizing productivity on Mac OS!
Ask episode
AI Snips
Chapters
Transcript
Episode notes
ANECDOTE

Past Scrapers

  • Wes built a scraper to notify him of PS5 availability and COVID vaccine appointments.
  • He also built marketplace scrapers for items like road bikes.
ANECDOTE

Cloud App Downloader

  • Wes created a scraper to download his 12,000 files from CloudApp (now Zyte).
  • He reverse-engineered their API using Proxyman.
ANECDOTE

Canadian Tire Scraper

  • Wes built a scraper for Canadian Tire's clearance items.
  • It finds deals over 90% off based on location.
Get the Snipd Podcast app to discover more snips from this episode
Get the app