To avoid unwanted content while in the research indexes, webmasters can instruct spiders to not crawl specified documents or directories from the common robots.txt file in the basis directory with the domain. Additionally, a webpage could be explicitly excluded from the search engine's database by making use of a meta https://news.rafeeg.ae