Jan 14, 2023 Unacknowledged SYNs by country
It’s sometimes interesting to look at how different a single day might be versus the longer-term trends. And to see what happens when you make changes to your pf rules.
I added all RU networks I was blocking from ssh to the list blocked for everything. I also fired up a torrent client on my desktop.
RU moving up the list versus the previous 5 days is no surprise; a good portion of traffic I receive from RU is port scanning. But I’ll have to look to see what caused the CZ numbers to climb.
I think the only interesting thing about the torrent client is that I should do something to track UDP in a similar manner as I track TCP. If I have a torrent client running, I will wind up with a lot of UDP traffic (much of it directed to port 6881 on this day), and will respond with ICMP port unreachable. To some extent this is a burden on my outbound bandwidth, but on the other hand it will allow me to add an easy new tracker to mcflowd: “to whom am I sending ICMP port unreachables?”. Of course, UDP is trivially spoofed, so I don’t truly know the source of the UDP.
Source IPV4 address space blocked from ssh, by country
Below is a chart showing the number of IPv4 addresses blocked from accessing ssh on my network, by country, for the 25 countries with the most address space blocked. It has changed over the years, but the U.S. is now #2 where it wasn’t in 2017 (it wasn’t even in the top 20 back then). What hasn’t changed: China remains dominant.
It’s worth noting that my automation blocks address space based on access attempts. A prefix doesn’t wind up in the list unless the automation has seen failed access attempts from that prefix. Of course, policy and address allocation determine the width of prefix, as well as aggregation. No one gets blocked forever, but repeat offenders are blocked longer. Bear in mind that ssh on my network is intended for a single user: me. If I’m not in China at the moment, ssh need not be accessible from China.
This next chart comes from mcflowd data. One of the things it tracks is unacknowledged SYN packets that hit my gateway, per source IPV4 address. I can process this data with IP to country information to get an idea of which countries are the predominant prowlers. It shows how often different countries are either probing for a service I don’t run or trying to hit a service that I’ve blocked some of their IPv4 address space from accessing. A metaphor: the number of times they knocked on my door and I purposely didn’t answer (I did not reply with a SYN ACK). I call this the chump factor chart; many of the address spaces that contribute to this chart have been the same for 5 years.
Note that this second chart is from a period of less than 5 days. Anyone with a public Internet connection should not kid themselves into thinking they’re not being probed, constantly.
This is how you splinter the Internet. Make us fed up enough with your traffic that’s indistinguishable from traffic with criminal intent, we block you. Works great for authoritarian governments that don’t want their citizenry to communicate with the free world, and those with other motives too. 🙁
The good news for me is that I have automation that’s pretty flexible in configuration and input sources (the log parser, for example, can be used on a number of different log formats as long as they’re text and contain offending IP addresses). It saves the data I need, and isn’t a significant resource consumer on my gateway. It’s very secure, using my Credence library for ECDH, authentication and authorization (which under the hood is using libsodium at the moment). I have a reasonably robust IP to country service, which updates itself via RDAP. Sadly the registries are a disaster saga, so occasionally I wind up reloading with GeoLite or similar data. But since I only use the country data to determine how long and how wide a prefix will be blocked, and not if a prefix will be blocked, it’s mostly inconsequential. It’s just useful to be able to see where the nefarious traffic is coming from, through a geopolitical lens.
Hey U.S. (my own country): we shouldn’t throw stones while we live in a glass house. And if there’s anything we should do about big tech, I’d say regulating the weaponization of massive cloud computing resources would be a good start. Where do a lot of the U.S. probes come from? Amazon EC2, Google, Microsoft, DigitalOcean, linode, Oracle. The same holds true for probes from Canada and other parts of the western world. Some of these are legitimate research probes. However, to a large extent they’re indistinguishable from nefarious activity. And besides, I pay for my bandwidth at the end of a thin straw we call broadband in the U.S. I don’t want this traffic, yet I pay for it.
mcblockd 5 years on
mcblockd, the firewall automation I created 5 years ago, continues to work.
However, it’s interesting to note how things have changed. Looking at just the addresses I block from accessing port 22…
While China remains at the top of my list of total number of blocked IP addresses, the US is now in 2nd place. In 2017, the US wasn’t even in the top 20. What has changed?
Most of the change here is driven by my automation seeing more and more attacks originating from cloud hosted services. Amazon EC2, Google, Microsoft, DigitalOcean, Linode, Oracle, et. al. While my automation policy won’t go wider than a /24 for a probe from a known US entity, over time I see probes from entire swaths of contiguous /24 networks from the same address space allocation, which will be coalesced to reduce firewall table size. Two adjacent /24 networks become a single /23. Two adjacent /23 networks become a single /22. All the way up to a possible /8 (the automation stops there).
So today, the last of 2022, I see some very large blocks owned by our cloud providers being blocked by my automation due to receiving ssh probes from large contiguous swaths of their address space.
I am very appreciative of the good things from big tech. But I’m starting to see the current cloud computing companies as the arms dealers of cyberspace.
My top 2 countries:
However, it’s interesting to note how things have changed. Looking at just the addresses I block from accessing port 22…
While China remains at the top of my list of total number of blocked IP addresses, the US is now in 2nd place. In 2017, the US wasn’t even in the top 20. What has changed?
Most of the change here is driven by my automation seeing more and more attacks originating from cloud hosted services. Amazon EC2, Google, Microsoft, DigitalOcean, Linode, Oracle, et. al. While my automation policy won’t go wider than a /24 for a probe from a known US entity, over time I see probes from entire swaths of contiguous /24 networks from the same address space allocation, which will be coalesced to reduce firewall table size. Two adjacent /24 networks become a single /23. Two adjacent /23 networks become a single /22. All the way up to a possible /8 (the automation stops there).
So today, the last of 2022, I see some very large blocks owned by our cloud providers being blocked by my automation due to receiving ssh probes from large contiguous swaths of their address space.
I am very appreciative of the good things from big tech. But I’m starting to see the current cloud computing companies as the arms dealers of cyberspace.
My top 2 countries:
CN 131,560,960 addresses
/9 networks: 1 (8,388,608 addresses)
/10 networks: 10 (41,943,040 addresses)
/11 networks: 12 (25,165,824 addresses)
/12 networks: 18 (18,874,368 addresses)
/13 networks: 29 (15,204,352 addresses)
/14 networks: 48 (12,582,912 addresses)
/15 networks: 48 (6,291,456 addresses)
/16 networks: 37 (2,424,832 addresses)
/17 networks: 14 (458,752 addresses)
/18 networks: 7 (114,688 addresses)
/19 networks: 10 (81,920 addresses)
/20 networks: 5 (20,480 addresses)
/21 networks: 3 (6,144 addresses)
/22 networks: 3 (3,072 addresses)
/23 networks: 1 (512 addresses)
US 92,199,996 addresses
/9 networks: 3 (25,165,824 addresses)
/10 networks: 5 (20,971,520 addresses)
/11 networks: 10 (20,971,520 addresses)
/12 networks: 9 (9,437,184 addresses)
/13 networks: 16 (8,388,608 addresses)
/14 networks: 10 (2,621,440 addresses)
/15 networks: 8 (1,048,576 addresses)
/16 networks: 42 (2,752,512 addresses)
/17 networks: 10 (327,680 addresses)
/18 networks: 11 (180,224 addresses)
/19 networks: 8 (65,536 addresses)
/20 networks: 10 (40,960 addresses)
/21 networks: 2 (4,096 addresses)
/22 networks: 9 (9,216 addresses)
/23 networks: 9 (4,608 addresses)
/24 networks: 818 (209,408 addresses)
/25 networks: 4 (512 addresses)
/26 networks: 5 (320 addresses)
/27 networks: 5 (160 addresses)
/28 networks: 2 (32 addresses)
/29 networks: 7 (56 addresses)
/30 networks: 1 (4 addresses)
You can clearly see the effect of my automation policy for the US. Lots of /24 networks get added, most of them with a 30 to 35 day expiration. Note that expirations increase for repeat offenses. But over time, as contiguous /24 networks are added due to sending probes at my firewall, aggregation will lead to wider net masks (shorter prefix lengths). Since I’m sorting countries based on the total number of addresses I’m blocking, obviously shorter prefixes have a much more profound effect than long prefixes.
libDwmCredence integrated into mcrover
I’ve completed my first pass at libDwmCredence (a replacement for libDwmAuth). As previously mentioned, I did this work in order to drop my use of Crypto++ in some of my infrastructure. Namely mcrover, my personal host and network monitoring software.
I’ve integrated it into mcrover, and deployed the new mcrover on 6 of my internal hosts. So far, so good.
mcperf: a multithreaded bandwidth tester
I’ve been really dismayed by the lack of decent simple tools for testing the available bandwidth between a pair of hosts above 1 gigabit/second. Back when I didn’t have any 10 gigabit connections at home, I used iperf and iperf3. But I now have several 10 gigabit connections on my home network, and since these tools don’t use multithreading effectively, they become CPU bound (on a single core) before they reach the target bandwidth. Tools like ssh and scp have the same problem; they’re single threaded and become CPU bound long before they saturate a 10 gigabit connection.
When I install a 10 gigabit connection, whether it’s via SFP+ DACs, SFP+ SR optics or 10GbaseT, it’s important that I’m able to test the connection’s ability to sustain somewhere near line rate transfers end-to-end. Especially when I’m buying my DACs, transceivers or shielded cat6a patch cables from eBay or any truly inexpensive vendor. I needed a tool that could saturate a 10 gigabit connection and report the data transfer rate at the application level. Obviously due to the additional data for protocol headers and link encapsulation, this number will be lower than the link-level bandwidth, but it’s the number that ultimately matters for an application.
So, I quickly hacked together a multithreaded application to test my connections at home. It will spawn the requested number of threads (on each end) and the server will send data from each thread. Each thread gets its own TCP connection.
For a quick hack, it works well.
dwm@www:/home/dwm% mcperf -t 4 -c kiva
bandwidth: 8.531 Gbits/sec
bandwidth: 8.922 Gbits/sec
bandwidth: 9.069 Gbits/sec
bandwidth: 9.148 Gbits/sec
bandwidth: 9.197 Gbits/sec
bandwidth: 9.230 Gbits/sec
bandwidth: 9.253 Gbits/sec
bandwidth: 9.269 Gbits/sec
bandwidth: 9.283 Gbits/sec
Given that I don’t create servers that don’t use strong authentication, even if they’ll only be run for 10 seconds, I’m using the PeerAuthenticator from libDwmAuth for authentication. No encryption of the data that’s being sent, since it’s not necessary.
Of course this got me thinking about the number of tools we have today that just don’t cut it in a 10 gigabit network. ssh, scp, ftp, fetch, etc. Even NFS code has trouble saturating a 10 gigabit connection. It seems like eons ago that Herb Sutter wrote “The Free Lunch Is Over”. It was published in 2005. Yet we still have a bunch of tools that are CPU bound due to being single-threaded. How are we supposed to take full advantage of 10 gigabit and faster networks if the tools we use for file transfer, streaming, etc. are single-threaded and hence CPU bound well before they reach 10 gigabits/second? What happens when I run some fiber at home for NAS and want to run 40 gigabit or (egads!) 100 gigabit? It’s not as if I don’t have the CPU to do 40 gigabits/second; my NAS has 12 cores and 24 threads. But if an application is single-threaded, it becomes CPU bound at around 3.5 gigabits/second on a typical server CPU core. 🙁 Sure, that’s better than 1 gigabit/second but it’s less than what a single SATA SSD can do, and much less than what an NVME/M.2/striped SATA SSD/et. al. can do.
We need tools that aren’t written as if it’s 1999. I suspect that after I polish up mcperf a little bit, I’m going to work on my own replacement for scp so I can at least transfer files without being CPU bound at well below my network bandwidth.
short flurry of ssh login attempts blocked by mcblockd
mcblockd added quite a few networks during a 20 minute period today. I don’t have an explanation for the ssh login attempts all coming in during this period, but it’s nice to see that mcblockd happily blocked all of them.
While this is by no means a high rate of attempts, it’s higher than what I normally see.
May 22 11:32:10 ria mcblockd: [I] Added 185.129.60/22 (DK) to ssh_losers for 180 days May 22 11:32:11 ria mcblockd: [I] Added 89.234.152/21 (FR) to ssh_losers for 180 days May 22 11:32:45 ria mcblockd: [I] Added 46.233.0/18 (BG) to ssh_losers for 180 days May 22 11:33:00 ria mcblockd: [I] Added 216.218.222/24 (US) to ssh_losers for 30 days May 22 11:33:05 ria mcblockd: [I] Added 199.87.154/24 (CA) to ssh_losers for 30 days May 22 11:33:15 ria mcblockd: [I] Added 78.109.16/20 (UA) to ssh_losers for 180 days May 22 11:33:18 ria mcblockd: [I] Added 89.38.148/22 (FR) to ssh_losers for 180 days May 22 11:33:26 ria mcblockd: [I] Added 65.19.167/24 (US) to ssh_losers for 30 days May 22 11:34:05 ria mcblockd: [I] Added 62.212.64/19 (NL) to ssh_losers for 180 days May 22 11:35:54 ria mcblockd: [I] Added 190.10.0/17 (CR) to ssh_losers for 180 days May 22 11:37:16 ria mcblockd: [I] Added 192.42.116/22 (NL) to ssh_losers for 180 days May 22 11:38:33 ria mcblockd: [I] Added 199.249.223/24 (US) to ssh_losers for 30 days May 22 11:38:37 ria mcblockd: [I] Added 173.254.216/24 (US) to ssh_losers for 30 days May 22 11:39:48 ria mcblockd: [I] Added 128.52.128/24 (US) to ssh_losers for 30 days May 22 11:39:51 ria mcblockd: [I] Added 64.113.32/24 (US) to ssh_losers for 30 days May 22 11:40:32 ria mcblockd: [I] Added 23.92.27/24 (US) to ssh_losers for 30 days May 22 11:40:50 ria mcblockd: [I] Added 162.221.202/24 (CA) to ssh_losers for 30 days May 22 11:42:42 ria mcblockd: [I] Added 91.213.8/24 (UA) to ssh_losers for 180 days May 22 11:43:37 ria mcblockd: [I] Added 162.247.72/24 (US) to ssh_losers for 30 days May 22 11:44:34 ria mcblockd: [I] Added 193.110.157/24 (NL) to ssh_losers for 180 days May 22 11:44:38 ria mcblockd: [I] Added 128.127.104/23 (SE) to ssh_losers for 180 days May 22 11:45:50 ria mcblockd: [I] Added 179.43.128/18 (CH) to ssh_losers for 180 days May 22 11:45:55 ria mcblockd: [I] Added 89.144.0/18 (DE) to ssh_losers for 180 days May 22 11:46:29 ria mcblockd: [I] Added 197.231.220/22 (LR) to ssh_losers for 180 days May 22 11:46:44 ria mcblockd: [I] Added 195.254.132/22 (RO) to ssh_losers for 180 days May 22 11:46:54 ria mcblockd: [I] Added 154.16.244/24 (US) to ssh_losers for 30 days May 22 11:47:52 ria mcblockd: [I] Added 87.118.64/18 (DE) to ssh_losers for 180 days May 22 11:48:51 ria mcblockd: [I] Added 46.165.224/19 (DE) to ssh_losers for 180 days May 22 11:50:13 ria mcblockd: [I] Added 178.17.168/21 (MD) to ssh_losers for 180 days May 22 11:50:47 ria mcblockd: [I] Added 31.41.216/21 (UA) to ssh_losers for 180 days May 22 11:50:55 ria mcblockd: [I] Added 62.102.144/21 (SE) to ssh_losers for 180 days May 22 11:51:19 ria mcblockd: [I] Added 64.137.244/24 (CA) to ssh_losers for 30 days May 22 11:52:28 ria mcblockd: [I] Added 80.244.80/20 (SE) to ssh_losers for 180 days May 22 11:52:42 ria mcblockd: [I] Added 192.160.102/24 (CA) to ssh_losers for 30 days May 22 11:53:06 ria mcblockd: [I] Added 176.10.96/19 (CH) to ssh_losers for 180 days May 22 11:55:38 ria mcblockd: [I] Added 77.248/14 (NL) to ssh_losers for 180 days May 22 11:56:20 ria mcblockd: [I] Added 199.119.112/24 (US) to ssh_losers for 30 days May 22 11:56:32 ria mcblockd: [I] Added 94.142.240/21 (NL) to ssh_losers for 180 days
China is a lousy netizen
There’s no one even close in terms of ssh login attempts. In a span of two weeks, mcblockd has blocked 47 million more addresses from China. That doesn’t mean I’ve seen 47 million IP addresses in login attempts. It means that China has a lot of address space being used to probe U.S. sites.
Brazil is in second place, but they’re behind by more than a decimal order of magnitude. Below are the current top two countries being blocked by mcblockd, by quantity of address space.
% mcblockc getactive ssh_losers
...
Addresses covered per country:
CN 149,911,680
/10 networks: 10 (41,943,040 addresses)
/11 networks: 21 (44,040,192 addresses)
/12 networks: 38 (39,845,888 addresses)
/13 networks: 26 (13,631,488 addresses)
/14 networks: 23 (6,029,312 addresses)
/15 networks: 26 (3,407,872 addresses)
/16 networks: 14 (917,504 addresses)
/17 networks: 4 (131,072 addresses)
/18 networks: 1 (16,384 addresses)
/19 networks: 1 (8,192 addresses)
/21 networks: 2 (4,096 addresses)
/22 networks: 2 (2,048 addresses)
/25 networks: 1 (128 addresses)
BR 14,170,112
/10 networks: 1 (4,194,304 addresses)
/11 networks: 3 (6,291,456 addresses)
/12 networks: 1 (1,048,576 addresses)
/13 networks: 3 (1,572,864 addresses)
/14 networks: 3 (786,432 addresses)
/15 networks: 1 (131,072 addresses)
/17 networks: 2 (65,536 addresses)
/18 networks: 1 (16,384 addresses)
/19 networks: 5 (40,960 addresses)
/20 networks: 2 (8,192 addresses)
/21 networks: 5 (10,240 addresses)
/22 networks: 4 (4,096 addresses)
I seriously doubt that Chinese citizens have anything to do with these attempts. I’m told that the Great Firewall blocks most ssh traffic on port 22. Not to mention that China’s Internet connectivity is somewhere near 95th in the world in terms of available bandwidth, so it’d be terribly painful for an ordinary user to use ssh or scp from China to my gateway. I think I can assume this is all government-sponsored probing.
mcblockd’s latest trick works: drop TCP connections
Evidence in the logs of mcblockd’s latest feature working. It’s successfully killing TCP connections when it adds a prefix to one of the pf tables.
Apr 29 03:42:40 ria mcblockd: [I] Dropped TCP connection from 221.144.5.116:38440 Apr 29 03:42:40 ria mcblockd: [I] Added 221.144/12 (KR) to ssh_losers for 180 days Apr 29 05:02:02 ria mcblockd: [I] Dropped TCP connection from 46.118.248.195:40294 Apr 29 05:02:02 ria mcblockd: [I] Added 46.118/15 (UA) to ssh_losers for 180 days Apr 29 07:07:42 ria mcblockd: [I] Dropped TCP connection from 120.132.4.45:56388 Apr 29 07:07:42 ria mcblockd: [I] Added 120.128/13 (CN) to ssh_losers for 180 days Apr 29 10:04:23 ria mcblockd: [I] Dropped TCP connection from 95.215.2.52:50862 Apr 29 10:04:23 ria mcblockd: [I] Added 95.215.0/22 (RU) to ssh_losers for 180 days Apr 29 11:51:34 ria mcblockd: [I] Dropped TCP connection from 110.246.84.64:32309 Apr 29 11:51:34 ria mcblockd: [I] Added 110.240/12 (CN) to ssh_losers for 180 days Apr 29 12:22:42 ria mcblockd: [I] Dropped TCP connection from 183.184.133.58:3369 Apr 29 12:22:42 ria mcblockd: [I] Added 183.184/13 (CN) to ssh_losers for 180 days Apr 29 13:13:54 ria mcblockd: [I] Dropped TCP connection from 120.150.231.99:50357 Apr 29 13:13:54 ria mcblockd: [I] Dropped TCP connection from 120.150.231.99:50349 Apr 29 13:13:54 ria mcblockd: [I] Added 120.144/12 (AU) to ssh_losers for 180 days Apr 29 14:42:30 ria mcblockd: [I] Dropped TCP connection from 113.209.68.135:53280 Apr 29 14:42:30 ria mcblockd: [I] Added 113.209/16 (CN) to ssh_losers for 180 days
mcblockd’s latest tricks: kill pf state, walk PCB list and kill TCP connections
Today I added a new feature to mcblockd to kill pf state for all hosts in a prefix when the prefix is added to one of my pf tables. This isn’t exactly what I want, but it’ll do for now.
mcblockd also now walks the PCB (protocol control block) list and drops TCP connections for hosts in a prefix I’ve just added to a pf table. Fortunately there was sample code in /usr/src/usr.sbin/tcpdrop/tcpdrop.c. The trick here is that I don’t currently have a means of mapping a pf table to where it’s applied (which ports, which interfaces). In the long term I might add code to figure that out, but in the interim I can configure ports and interfaces in mcblockd’s configuration file that will allow me to drop specific connections. For this first pass, I just toast all PCBs for a prefix.
The reason I added this feature: I occasionally see simultaneous login attempts from different IP addresses in the same prefix. If I’m going to block the prefix automatically, I want to cut off all of their connections right now, not after all of their connections have ended. Blowing away their pf state works, but leaves a hanging TCP connection in the ESTABLISHED state for a while. I want the PCBs to be cleaned up.