By Luis Rijo
Publication Date: 2026-10-31 20:32:00
Gary Illyes of Google said it would take him “probably 30 seconds” to set up a script that crawls the web looking like Googlebot, in a podcast interview published on October 28, 2026, in which he described catching impostors with honeypot websites, called the web’s main crawler-control file “a suggestion” and explained why Google has started signing some requests from its AI agents with cryptographic keys.
In Short
Any program that visits a website can claim to be Google’s crawler just by writing Google’s name on its request, and a Google employee who spent years on its crawler team says setting that up takes about 30 seconds. That matters because many sites wave Google’s crawler through, while a fake one might copy content for unknown purposes or hit a site so hard that real visitors can’t load it. The only reliable check today is whether the visit comes from internet addresses Google publishes, and Google is testing digital signatures on some requests that a forger cannot…


