I still have two /24 Google networks blocked by automation: 66.249.66.0/24 and 66.249.72.0/24. These are used by Googlebot crawlers. I don’t intend to leave them blocked indefinitely. But with just these two /24 networks, Google sits at the top of the list of TCP SYN senders I don’t acknowledge.
I don’t log User-agent on my web server, nor use it to make any decisions. Some would argue that I could use it to differentiate Google Gemini / Vertex AI scraping versus other Google bots. However, User-agent isn’t reliable. I’ve seen rogue crawlers with User-agent set to Googlebot or some other bot they want to impersonate for some reason. My favorite values are the ones with typos. 🙂 In the age of LLM training, the longstanding handshake agreement is breaking down. For the same reason, I don’t use robots.txt to try to prevent crawlers from crawling particular content. I can’t trust a web client to abide a handshake agreement that’s trivially betrayed and unenforceable. I don’t expect Google to violate the handshake agreement. But if I want to block or not block Google bots, I’m going to do it via more reliable means than User-agent.
For this reason, I wish Google would explicitly use different address space for different bots. It would make blocking or shaping traffic to/from their bots easier and more reliable.