Category: Web Development

I recently posted about doing my annual web log perusal and adding more networks to the list which I block from accessing my web server. Until 2021 or so, most of the networks I added were what I’d consider hostile netizens, and most of them in foreign countries (China far and away the worst offender, followed by Russia, Singapore, Hong Kong and others).

Around 2021 (maybe earlier), I started seeing more attacks coming from cloud infrastructure. Today, after my first full pass of changes (March 17, 2024), it’s the majority.

As an aside, how do I know that many of the adversaries are foreign? Well, after my first pass of changes, it’s pretty clear that a lot of them were far away (by network round-trip time). Below are plots of traffic and round trip times to clients of my web server, from before my changes and right after my changes. Note how the 95th and 75th percentile round trip times dropped dramatically. That’s because many of the nefarious users were far away.

It’s worth noting that much of my traffic is local to my home. I have many Raspberry Pis and other hosts running my ‘mcrover’ software, and one of the things it does it monitor my web server, including queries to my blog and the gallery software for Randy’s site since they have database backends I want to be sure are running. This background monitoring traffic works out to about 80 kilobits/second on average. The round trip time for this traffic is tiny; for some hosts it’s sub-millisecond since they have 10 gigabit connections (DAC, fiber or 10GbaseT).

Back to the topic at hand…

Late last year I got a question via email from a customer at DigitalOcean as to why I had blocked their access. I didn’t reply. I’ve come to the conclusion that if users won’t police themselves, and their cloud provider won’t police their users (and expects us to help them do the policing!), neither deserve my time. If you, or your customer, are getting 404’s from my website for URLs like ‘/login.php’ (which doesn’t exist and has never existed on my site), you’re either a criminal or an accomplice. Neither of you deserve my time. Your activities are hostile, period.

Since 2021 or so, the more disturbing trend I see seems to be driven by the AI arms race in big tech. Microsoft (and OpenAI), Google and Apple crawling my web site with more frequency, and pulling everything they can find. PDFs, all images, etc. There was a time when I considered it OK, in the sense that it was used only for search purposes, which at least gives back a bit to the web as a whole (search is a useful service, despite the mess it has become). But today, it’s pretty clear to me that the snarfing of data by big tech is not balanced by a benefit to users of the web. It’s also quite aggressive. How do I benefit from Microsoft downloading every PDF and image that’s on my web site? I don’t. I lose (I pay for my bandwidth), they profit. There is literally zero benefit to me, or you unless you’re Microsoft or OpenAI.

I’ve grown tired of this. I’m not interested in feeding the LLMs, much less allowing cloud infrastructure to be leveraged against my web site. So as of this week, anytime I happen to be doing some block list maintenance and see traffic from a cloud service or big tech, the network will be blocked. And not just small chunks; the entire CIDR allocation. And not just for a short period of time; 2 years. If you manage to get your act together 2 years from now, great. If not, you’ll be blocked for 2 more years. I anticipate the latter, since that’s been the steady course for the last 7 years.

I’ve already blocked large swaths of Amazon, Microsoft, Google, DigitalOcean, Linode, Hetzner, Hurricane Electric, OVH and other cloud services. There is literally no reason I should see requests from any of these address spaces. They’re not human beings reading my web pages. They are nefarious automation, and not policed by the cloud providers. We are the victims. Ideally the cloud providers would be using deep packet inspection and proactively shut down such activities. But if no one is going to jail for their paying customers’ illicit activities… nothing gets done. And most of us don’t have time to do the policing for them. It’s MUCH faster for me to just block the cloud provider’s entire address space. I’m not going to play wack-a-mole, nor serve as their unpaid reporter of nefarious activity they’re enabling. A simple buh-bye to the cloud provider and all of their customers is my far and away best option. This isn’t a rash decision on my part; I’ve seen nefarious traffic from the cloud providers for many years, and given that they hold gobs of address space, it’s futile to block only small parts. The only sane course is to block all of their address space. In particular, I’m talking to you Amazon, Google, DigitalOcean, Hetzner, Linode, Microsoft and OVH.

It is highly likely that I’ll soon be blocking crawler address space belonging to Google, Apple, Microsoft and others. Sad but true that the benefit of being indexed by search engines has finally been eclipsed by the downsides of allowing unfettered access from big tech. robots.txt doesn’t get us there because it doesn’t have the granularity we need, and the reality is that many of the crawlers don’t even bother to look at it. Worse, how are we as web site admins supposed to be able to determine how the data is being used? If the request is coming from Google crawler address space, is it for search or for feeding their LLMs? Or for sale to a third party? Or for driving their ad revenue? The reality is that we don’t know, and have no means of knowing. I’d rather my web site disappear from search than have all of my web site data used against me in some manner (which includes directed advertising). And I’m tired of the amount of bandwidth it consumes. Google alone is responsible for over 3,000 downloads from my web site per week. Honestly, it’s pretty disgusting. My site is a TINY personal web site, mostly for my own use. I can’t imagine the barrage of traffic from Google to large web sites.

If you happen to be a victim of the more aggressive blocking I’m about to start and the automation that will maintain it long term, my apologies. But I’m unlikely to be sympathetic to pleas for access. If your neighbor attacks my site and my automation blocks the network you share with them as a result… I am truly sorry, but I don’t have time to poke tiny holes as exceptions. I’ll likely leave the 30 day with geometric escalation policy in place for U.S. broadband users, but if you’re using a VPN through a cloud provider and someone else uses that provider to abuse my web site… you’ll be out of luck for much longer. Again, my apologies. If you want to do something about what’s been happening for years on this front that’s forcing some of us to block large swaths of cloud provider space and probably soon the web crawlers of Google, Microsoft, Apple and others… reach out to your legislators. If you’re in the U.S., maybe just making them aware that cloud services on U.S. soil are being used every minute of every day to attack law abiding, tax paying citizens who just want to share knowledge with other citizens. Good luck, godspeed, live long and prosper.