I still have two \/24 Google networks blocked by automation: 66.249.66.0\/24 and 66.249.72.0\/24. These are used by Googlebot crawlers. I don’t intend to leave them blocked indefinitely. But with just these two \/24 networks, Google sits at the top of the list of TCP SYN senders I don’t acknowledge.<\/p>\n\n\n\n
I don’t log User-agent on my web server, nor use it to make any decisions. Some would argue that I could use it to differentiate Google Gemini \/ Vertex AI scraping versus other Google bots. However, User-agent isn’t reliable. I’ve seen rogue crawlers with User-agent set to Googlebot or some other bot they want to impersonate for some reason. My favorite values are the ones with typos. \ud83d\ude42 In the age of LLM training, the longstanding handshake agreement is breaking down. For the same reason, I don’t use robots.txt to try to prevent crawlers from crawling particular content. I can’t trust a web client to abide a handshake agreement that’s trivially betrayed and unenforceable. I don’t expect Google to violate the handshake agreement. But if I want to block or not block Google bots, I’m going to do it via more reliable means than User-agent.<\/p>\n\n\n\n
For this reason, I wish Google would explicitly use different address space for different bots. It would make blocking or shaping traffic to\/from their bots easier and more reliable.<\/p>\n\n\n\n