Search engines like yahoo generally cache the contents in the robots.txt so that they don’t will need to help keep downloading it, but will usually refresh it quite a few times daily. Which means that modifications to instructions are typically reflected quite quickly. A robots.txt file helps prevent search engines https://rankingseotoolz.com/robots-txt-generator