3min chapter

Software Engineering Radio - the podcast for professional software developers cover image

Episode 503: Diarmuid McDonnell on Web Scraping

Software Engineering Radio - the podcast for professional software developers

CHAPTER

How to Scrape Non Profits

In research in academia it's less about crawling web pages to see where the connections lie, though sometimes that may be of interest. It's more about collecting specific information on the web page that goes on to help you answer your research question. Togan: Usually the process is to request a data file to one of the publicly available data files. Request that i open it up in python and extract unique ideas for all of the non profits. So then that becomes a separate process for each regulator, cycling through those list collecting all f the information i need. And then the final stage essentially is packaging all f those up into a single data set as well.

00:00

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode