How Does a Search Engine Crawl the Web?

A crawler has to go through each web page, look at all of its links. And keep following links until there's no more links to follow. Crawling is happening even if I'm not searching for something like immediately. Now, a lot of this information is also proprietary, so we don't know exactly how every search engine works. We can surmise some of the details, but we're not never going to know all the details. That's what makes Google so good is a lot of its proprietary algorithms.