In order to avoid unwanted content material in the search indexes, site owners can instruct spiders to not crawl particular documents or directories from the regular robots.txt file in the root directory on the area. Furthermore, a web page can be explicitly excluded from the search engine's databases by using a meta tag precise to robots (normally