Robots.txt is usually a file that exists on the basis Listing of every Web page and can be used to instruct search engines like google and yahoo on which directories/information of the web site they can crawl and incorporate in their index. Should you be expecting them for being 100% https://news.rafeeg.ae