The Web Crawler

The web was growing quickly in the ‘90s. But all that growth wasn’t going to lead to much if people couldn’t actually find any web sites. In 1995, an innovative new tool started crawling the web. And the search engine it fed opened the doors to the World Wide Web. 

Elizabeth Van Couvering describes trying to find websites before search engines, and how difficult it was becoming in the early ’90s to keep track of them all. Louis Monier talks about having to convince others how important search engines would become—and he showed them what a web crawler could do. Paul Cormier recounts taking the search engine from a research project to a commercial one. And Richard Seltzer wrote the book on search engines, helping the rest of the world see what a profoundly vital tool they would become.
 

If you want to read up on some of our research on search, you can check out all our bonus material over at redhat.com/commandlineheroes. The page is built in the style of 1995—check it out.

Follow along with the episode transcript.