The Challenges of Web Scraping

There is a distinction between those different levels of access to pages. Often if information is accessible via web scraping, then that usually does not apply to information held behind authentication. So private pages, members only areas, they're usually restricted from your web scraping activities and often for good reason. And it's not something i've ever tried to overcome, though there are technical means of doing so.