Google’s John Mueller said that since the robots.txt file is cached by Google for about 24-hours, it does not make much sense to dynamically update your robots.txt file throughout the day to control controlling.
Google won’t…
Article Source
https://www.seroundtable.com/google-robots-txt-24-hours-38763.html



