The Use Cases for Web Scraping

i'l begin with mine. As an academic and as a researcher, i'm interested in a large scale administrative data about non profits around the world. I need to know who sits on the board of these izations. So that led me to develop a reasonably simple web scraping application for australia. There're some common approaches and an techniques i'm sure we'll get into. But one particular challenge was the regulators web site does have an idea of who's making requests for their web pages. And i haven't counted exactly, but every one or two thousand requests, it would block that ip address. That meant that every couple of hundred requests, i would send my web scraping application