The Joe Reis Show Sarah McKenna - Web Scraping Explained
Apr 29, 2025
Sarah McKenna, CEO of Sequentum and a leader in web scraping standards, dives deep into the world of data collection. She explains web scraping’s diverse applications in finance and NGOs, and its evolution alongside alternative data trends. Privacy concerns and challenges like bot blocking are examined, highlighting the importance of ethical data practices. McKenna also explores how AI is transforming web scraping, enabling smaller teams and introducing new challenges in data quality. Get insights on starting your own web scraping endeavors!
AI Snips
Chapters
Transcript
Episode notes
Broad Applications of Web Scraping
- Web scraping extracts real-time or compiled data from the open web by either crawling or targeted scraping.
- It supports applications from tracking Tesla stations to identifying hate speech and fighting human trafficking.
Sarah's Web Scraping Origin Story
- Sarah McKenna was recruited into web scraping from automated quality assurance background.
- She enhanced a scraping business blocked by bot defenses by unblocking it within weeks using disciplined automation.
Understanding Alternative Data
- The term "alt data" originated in finance to mean non-traditional data sources that provide investment signals.
- It includes web data, weather, transaction and app usage data, not just traditional financial reports.