#8 How do Web Search Engines Work?
Kopec Explains Software
00:00
How Do I Get My Website to Not Be Crawled?
There's a standard for specifying pages that you want on your site to be crawled by a bot. There are some guidelines put out by Google to help you make your page properly indexed. So what an index acts as later on is when you go and do a search, how do we know the terms that you've searched for, which pages are actually associated with those terms? That's the purpose of the index.
Transcript
Play full episode
