Author: dwm

I still have two /24 Google networks blocked by automation: 66.249.66.0/24 and 66.249.72.0/24. These are used by Googlebot crawlers. I don’t intend to leave them blocked indefinitely. But with just these two /24 networks, Google sits at the top of the list of TCP SYN senders I don’t acknowledge.

I don’t log User-agent on my web server, nor use it to make any decisions. Some would argue that I could use it to differentiate Google Gemini / Vertex AI scraping versus other Google bots. However, User-agent isn’t reliable. I’ve seen rogue crawlers with User-agent set to Googlebot or some other bot they want to impersonate for some reason. My favorite values are the ones with typos. 🙂 In the age of LLM training, the longstanding handshake agreement is breaking down. For the same reason, I don’t use robots.txt to try to prevent crawlers from crawling particular content. I can’t trust a web client to abide a handshake agreement that’s trivially betrayed and unenforceable. I don’t expect Google to violate the handshake agreement. But if I want to block or not block Google bots, I’m going to do it via more reliable means than User-agent.

For this reason, I wish Google would explicitly use different address space for different bots. It would make blocking or shaping traffic to/from their bots easier and more reliable.

The only interesting thing about this day… one of Google’s crawlers tripped one of my automatic blockers (by loading my blog login page). So 66.249.66/24 got blocked. And as I expect (and blogged about before), Google is far and away the leading consumer of my web site. Just their crawlers in 66.249.66/24 account for 9,177 connection attempts per day. Yes, some of the SYNs are retries. But just the same… if your web site is around for a while, it’ll get hammered by Google day in and day out.

I already know their crawlers are not particularly smart. For example, I’ve blocked them from pulling PDF files because they grab the same ones over and over (often in the same week), including PDFs that haven’t changed in over 10 years. So I’m going to leave the block in place for a while to see what happens. I suspect nothing; I don’t think the crawlers are smart enough to stop trying, despite getting nothing in response.

Another day, mostly the same stuff. The only significant difference from recent days: one douchebag on Comcast in Illinois (67.163.12.10).

The douchebag in Illinois…

  Source Addr  Destination Port     Pkts  Bytes
  ------------ -----------------  ------ ------
  67.163.12.10 http               1.448K 57.92K
  67.163.12.10 imaps              1.447K 57.88K
  67.163.12.10 https              1.443K 57.72K
  67.163.12.10 submission         1.433K 57.32K
  67.163.12.10 imap                1.43K  57.2K
  67.163.12.10 ssh                1.418K 56.72K
  67.163.12.10 2222               1.417K 56.68K

Different day, similar stuff. One thing that has changed here is that I’ve disabled port 587 (submission) on my gateway, since the only mail I care about receiving on my local mail server is generated locally. I also added to the list of those I block from IMAPS. So now the spammers and those attempting to relay show up, as well as those trying to access my IMAPS service for my local mail. Hurricane Electric is a primary offender here, as is G-Core Labs. All of the traffic from Redoubt Networks is for port 587; they’re apparently happy to host spammers.

The total count of unacknowledged SYNs received on this day: 37,611. On average, that’s one every 1.81 seconds. Put another way, about 26 per minute. The disheartening part: this is more than 8X the number of SYNs I accepted. 89.5% of the TCP connection attempts directed at my home network from the public Internet are rejected.

Think about that for a minute… almost 90% of attempted incoming TCP connections are unwanted garbage. This is the modern Internet. 🙁

If you’re a webmaster or content creator, it’s worth noting that my two leading offenders, Microsoft and Amazon, are both defendants in lawsuits involving copyright. The problem here is that big tech is scraping every web site they can for LLM and other AI training data, seemingly with little to no regard for copyright or attribution. See https://www.theregister.com/2024/03/13/nyt_hacking_response/ for The NY Times claim against Microsoft and OpenAI, and https://www.theregister.com/2024/04/22/ghaderi_v_amazon/ for a claim against Amazon. In today’s landscape, blocking just their address space isn’t going to stop them since they can buy hosting elsewhere just like the rest of us. But I’m coming to the conclusion that I shouldn’t make it easy for any of them.

Same stuff, different day. The top 25 ASes probing my home network despite the fact that they get nothing in response to the TCP SYNs they’re sending. Per usual, the top 3 spots are occupied by U.S. cloud providers.

On the topic of undesired traffic directed at my home network from the public internet, let’s look at unacknowledged SYN packets for the top offending autonomous systems (ASes) on a per-port basis, for just one day.

It’s worth noting a distinction between two classes of SYNs I don’t acknowledge: those directed at ports on which there is nothing listening and those directed at ports on which something is listening but from a source which I don’t acknowledge due to a history of nefarious activity. Nefarious activity to my web server, for example, might include looking for vulnerabilities I’ve never had (say ‘/login.php’ or ‘phpmyadminwhatever…’).

What’s interesting about looking at things from this perspective is just the different nature of probes from various host/cloud providers. For example, compare what comes from Cloudflare (just web traffic) to the skulduggery-only that comes from G-Core Labs S.A.. The difference is fairly astounding to find at this level of granularity (entire autonomous systems). And of course we see what we’d expect from Amazon and Google… mostly port 443 and 80 probes, but also scans of the entire 16-bit port range.

So let’s take a look…

First up, today’s leading offender: Microsoft. Most of the probing is directed at my web server.

8075 Microsoft Corporation (US)
Port	Packets
443 (https)	5372
80 (http)	936
587 (submission)	7
2375	3
102 (iso-tsap)	2
1521	1
2000	1
21 (ftp)	1
3306 (mysql)	1
5432 (postgresql)	1
5985	1
6379	1
9000	1

In second place we have Huawei. All of it directed at my web server.

In third place we have G-Core Labs S.A.. Most of it login attempts. I recommend blocking this AS entirely. All of the traffic I’ve received from them is nefarious in nature. I only recently added all of their address space to my ‘deny’ lists, but parts of it had been blocked by automation before.

In fourth place we have Amazon. Most of it is directed at my web server, but there are probes to every port.

In fifth place we have Cloudflare. All of their probes were directed at my web server.

13335 Cloudflare, Inc. (US)
Port	Packets
80 (http)	1460
443 (https)	406

In sixth place we have Brightspeed. All of their probes were directed at my web server.

In seventh place we have SEMrush. All of their probes were directed at my web server.

In eighth place we have Google cloud.

In ninth place we have Facebook. All of their probes were directed at my web server.

In 10th place we have PT Batanghari Baik Net. Pure skulduggery: ssh only.

I have this data, every day, for every AS, in 5 minute intervals. And then some, actually, since the data is at the IP address level and I can quickly and easily roll it up into per-AS data. And of course I can use it to make decisions about who I should block from my home network.

This week’s SYNners. The U.S. cloud providers continue to be the worst offenders. I don’t see this changing without some type of legislation. The AI arms race continues unabated.

Note that Apple and Google will show up high in this list as soon as I add their crawler address space to my blocked address space. Google likely at the top, and Apple likely in second.

It’s sad that even the little hosters are happy to host scumbags.

203.55.81.13 - - [28/Apr/2024:17:03:27 -0400] "GET /.env HTTP/1.1" 404 196

% mccurtain 203.55.81.13
203.55.81.0/24:
  216167 FR, SKOALI Skoali SAS

% mccurtain 216167
103.102.228.0/24
103.102.230.0/24
203.55.81.0/24

Skoali SAS now blocked from my web server.

I added more ASes to my block list due to a bot. It’s a real shame that so many of our hosting providers host this stuff. I’ve said goodbye to the following ASes:

• 35830 BTT Group Finance Ltd (GB)
• 14576 HOSTING-SOLUTIONS (US)
• 26548 PUREVOLTAGE-INC (US)
• 51765 Oy Crea Nova Hosting Solution Ltd (FI)

Here are the relevant log entries.

45.140.207.88 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
217.145.226.191 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
193.163.92.142 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
146.19.44.19 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
212.119.45.113 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
45.140.206.222 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
193.203.9.239 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
89.19.35.35 - - [28/Apr/2024:02:57:13 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
5.181.170.191 - - [28/Apr/2024:02:57:14 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
185.96.37.86 - - [28/Apr/2024:02:57:14 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
45.140.207.20 - - [28/Apr/2024:02:57:14 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
45.10.164.37 - - [28/Apr/2024:02:57:14 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196
93.177.117.72 - - [28/Apr/2024:02:57:14 -0400] "GET /+CSCOE+/logon.html HTTP/1.1" 404 196