Posted on

mcblockd 5 years on

mcblockd, the firewall automation I created 5 years ago, continues to work.

However, it’s interesting to note how things have changed. Looking at just the addresses I block from accessing port 22…

While China remains at the top of my list of total number of blocked IP addresses, the US is now in 2nd place. In 2017, the US wasn’t even in the top 20. What has changed?

Most of the change here is driven by my automation seeing more and more attacks originating from cloud hosted services. Amazon EC2, Google, Microsoft, DigitalOcean, Linode, Oracle, et. al. While my automation policy won’t go wider than a /24 for a probe from a known US entity, over time I see probes from entire swaths of contiguous /24 networks from the same address space allocation, which will be coalesced to reduce firewall table size. Two adjacent /24 networks become a single /23. Two adjacent /23 networks become a single /22. All the way up to a possible /8 (the automation stops there).

So today, the last of 2022, I see some very large blocks owned by our cloud providers being blocked by my automation due to receiving ssh probes from large contiguous swaths of their address space.

I am very appreciative of the good things from big tech. But I’m starting to see the current cloud computing companies as the arms dealers of cyberspace.

My top 2 countries:

    CN 131,560,960 addresses
       /9 networks:    1 (8,388,608 addresses)
      /10 networks:   10 (41,943,040 addresses)
      /11 networks:   12 (25,165,824 addresses)
      /12 networks:   18 (18,874,368 addresses)
      /13 networks:   29 (15,204,352 addresses)
      /14 networks:   48 (12,582,912 addresses)
      /15 networks:   48 (6,291,456 addresses)
      /16 networks:   37 (2,424,832 addresses)
      /17 networks:   14 (458,752 addresses)
      /18 networks:    7 (114,688 addresses)
      /19 networks:   10 (81,920 addresses)
      /20 networks:    5 (20,480 addresses)
      /21 networks:    3 (6,144 addresses)
      /22 networks:    3 (3,072 addresses)
      /23 networks:    1 (512 addresses)

    US 92,199,996 addresses
       /9 networks:    3 (25,165,824 addresses)
      /10 networks:    5 (20,971,520 addresses)
      /11 networks:   10 (20,971,520 addresses)
      /12 networks:    9 (9,437,184 addresses)
      /13 networks:   16 (8,388,608 addresses)
      /14 networks:   10 (2,621,440 addresses)
      /15 networks:    8 (1,048,576 addresses)
      /16 networks:   42 (2,752,512 addresses)
      /17 networks:   10 (327,680 addresses)
      /18 networks:   11 (180,224 addresses)
      /19 networks:    8 (65,536 addresses)
      /20 networks:   10 (40,960 addresses)
      /21 networks:    2 (4,096 addresses)
      /22 networks:    9 (9,216 addresses)
      /23 networks:    9 (4,608 addresses)
      /24 networks:  818 (209,408 addresses)
      /25 networks:    4 (512 addresses)
      /26 networks:    5 (320 addresses)
      /27 networks:    5 (160 addresses)
      /28 networks:    2 (32 addresses)
      /29 networks:    7 (56 addresses)
      /30 networks:    1 (4 addresses)

You can clearly see the effect of my automation policy for the US. Lots of /24 networks get added, most of them with a 30 to 35 day expiration. Note that expirations increase for repeat offenses. But over time, as contiguous /24 networks are added due to sending probes at my firewall, aggregation will lead to wider net masks (shorter prefix lengths). Since I’m sorting countries based on the total number of addresses I’m blocking, obviously shorter prefixes have a much more profound effect than long prefixes.

Posted on

UPS fiasco and mcrover to the rescue

I installed a new Eaton 5PX1500RT in my basement rack this week. I’d call it “planned, sort of…”. My last Powerware 5115 1U UPS went into an odd state which precipitated the new purchase. However, it was on my todo list to make this change.

I already own an Eaton 5PX1500RT, which I bought in 2019. I’ve been very happy with it. It’s in the basement rack, servicing a server, my gateway, ethernet switches and broadband modem. As is my desire, it is under 35% load.

The Powerware 5115 was servicing my storage server, and also under 35% load. This server has dual redundant 900W power supplies.

Installation of the new UPS… no big deal. install the ears, install the rack rails, rack the UPS.

Shut down the devices plugged into the old UPS, plug them in to the new UPS. Boot up, check each device.

Install the USB cable from the UPS to the computer that will monitor the state of the UPS. Install Network UPS Tools (nut) on that computer. Configure it, start it, check it.

This week, at this step things got… interesting.

I was monitoring the old Powerware 5115 from ‘ria’. ‘ria’ is a 1U SuperMicro server with a single Xeon E3-1270 V2. It has four 1G ethernet ports and a Mellanox 10G SFP+ card. Two USB ports. And a serial port which has been connected to the Powerware 5115 for… I don’t know, 8 years?

I can monitor the Eaton 5PX1500RT via a serial connection. However, USB is more modern, right? And the cables are less unwieldy (more wieldy). So I used the USB cable.

Trouble started here. The usbhid-ups driver did not reliably connect to the UPS. When it did, it took a long time (in excess of 5 seconds, an eternity in computing time). ‘ria’ is running FreeBSD 12.3-STABLE on bare metal.

I initially decided that I’d deal with it this weekend. Either go back to using a serial connection or try using a host other than ‘ria’. However…

I soon noticed long periods where mcrover was displaying alerts for many services on many hosts. Including alerts for local services, whose test traffic does not traverse the machine I touched (‘ria’). And big delays when using my web browser. Hmm…

Poking around, I seemed to only be able to reliably reproduce a network problem by pinging certain hosts with ICMPv4 from ria and observing periods where the round trip time would go from .05 milliseconds to 15 or 20 seconds. No packets lost, just periods with huge delays. These were all hosts on the same 10G ethernet network. ICMPv6 to the same hosts: no issues. Hmm…

I was eventually able to correlate (in my head) what I was seeing in the many mcrover alerts. On the surface, many didn’t involve ‘ria’. But under the hood they DO involve ‘ria’ simply because ‘ria’ is my primary name server. So, for example, tests that probe via both IPv6 and IPv4 might get the AAAA record but not the A record for the destination, or vice versa, or neither, or both. ‘ria’ is also the default route for these hosts. I honed in on the 10G ethernet interface on ‘ria’.

What did IPV4 versus IPv6 have to do with the problem? I don’t know without digging through kernel source. What was happening: essentially a network ‘pause’. Packets destined for ‘ria’ were not dropped, but queued for later delivery. As many as 20 seconds later! The solution? Unplug the USB cable for the UPS and kill usbhid-ups. In the FreeBSD kernel, is USB hoarding a lock shared with part of the network stack?

usbhid-ups works from another Supermicro server running the same version of FreeBSD. Different hardware (dual Xeon L5640). Same model of UPS with the same firmware.

This leads me to believe this isn’t really a lock issue. It’s more likely an interrupt routing issue. And I do remember that I had to add hw.acpi.sci.polarity="low" to /boot/loader.conf on ‘ria’ a while ago to avoid acpi0 interrupt storms (commented out recently with no observed consequence). What I don’t remember: what were all the issues I found that prompted me to add that line way back when?

Anyway… today’s lesson. Assume the last thing you changed has high probability of cause, even if there seems to be no sensible correlation. My experience this week: “Unplug the USB connection to the UPS and the 10G ethernet starts working again. Wait, what?!”.

And today’s thanks goes to mcrover. I might not have figured this out for considerably longer if I did not have alert information in my view. Being a comes-and-goes problem that only seemed to be reproducible between particular hosts using particular protocols might have made this a much more painful problem to troubleshoot without reliable status information on a dedicated display. Yes, it took some thinking and observing, and then some manual investigation and backtracking. But the whole time, I had a status display showing me what was observable. Nice!

Posted on

An ode to NSFNET and ANSnet: a simple NMS for home

A bit of history…

I started my computing career at NSFNET at the end of 1991. Which then became ANSnet. In those days, we had a home-brewed network monitoring system. I believe most/all of it was originally the brainchild of Bill Norton. Later there were several contributors; Linda Liebengood, myself, others. The important thing for today’s thoughts: it was named “rover”, and its user interface philosophy was simple but important: “Only show me actionable problems, and do it as quickly as possible.”

To understand this philosophy, you have to know something about the primary users: the network operators in the Network Operations Center (NOC). One of their many jobs was to observe problems, perform initial triage, and document their observations in a trouble ticket. From there they might fix the problem, escalate to network engineering, etc. But it wasn’t expected that we’d have some omniscient tool that could give them all of the data they (or anyone else) needed to resolve the problem. We expected everyone to use their brains, and we wanted our primary problem reporter to be fast and as clutter-free as possible.

For decades now, I’m spent a considerable amount of time working at home. Sometimes because I was officially telecommuting, at other times just because I love my work and burn midnight hours doing it. As a result, my home setup has become more complex over time. I have 10 gigabit ethernet throughout the house (some fiber, some Cat6A).  I have multiple 10 gigabit ethernet switches, all managed.  I have three rackmount computers in the basement that run 7×24.  I have ZFS pools on two of them, used for nightly backups of all networked machines, source code repository redundancy, Time Machine for my macOS machines, etc.  I run my own DHCP service, an internal DNS server, web servers, an internal mail server, my own automated security software to keep my pf tables current, Unifi, etc.  I have a handful of Raspberry Pis doing various things.  Then there’s all the other devices: desktop computers in my office, a networked laser printer, Roku, AppleTV, Android TV, Nest thermostat, Nest Protects, WiFi access points, laptops, tablet, phone, watch, Ooma, etc.  And the list grows over time.

Essentially, my home has become somewhat complex.  Without automation, I spend too much time checking the state of things or just being anxious about not having time to check everything at a reasonable frequency.  Are my ZFS pools all healthy?  Are all of my storage devices healthy?  Am I running out of storage space anywhere?  Is my DNS service working?  Is my DHCP server working?  My web server?  NFS working where I need it?  Is my Raspberry Pi garage door opener working?  Are my domains resolvable from the outside world?  Are the cloud services I use working?  Is my Internet connection down?  Is there a guest on my network?  A bandit on my network?  Is my printer alive?  Is my internal mail service working?  Are any of my UPS units running on battery?  Are there network services running that should not be?  What about the ones that should be, like sshd?

I needed a monitoring system that worked like rover; only show me actionable issues.  So I wrote my own, and named it “mcrover”.  It’s more of a host and service monitoring system than a network monitoring system, but it’s distributed and secure (using ed25519 stuff in libDwmAuth).  It’s modern C++, relatively easy to extend, and has some fun bits (ASCII art in the curses client when there are no alerts, for example).  Like the old Network Operations Center, I have a dedicated display in my office that only displays the mcrover Qt client, 24 hours a day.  Since most of the time there are no alerts to display, the Qt client toggles between a display of the next week’s forecast and a weather radar image when there are no alerts.  If there are alerts, the alert display will be shown instead, and will not go away until there are no alerts (or I click on the page switch in the UI).  The dedicated display is driven by a Raspberry Pi 4B running the Qt client from boot, using EGLFS (no X11).  The Raspberry Pi4 is powered via PoE.  It is also running the mcrover service, to monitor local services on the Pi as well as many network services.  In fact the mcrover service is running on every 7×24 general purpose computing device.  mcrover instances can exchange alerts, hence I only need to look at one instance to see what’s being reported by all instances.

This has alleviated me of a lot of sys admin and network admin drudgery.  It wasn’t trivial to implement, mostly due to the variety (not the quantity) of things it’s monitoring.  But it has proven itself very worthwhile.  I’ve been running it for many months now, and I no longer get anxious about not always keeping up with things like daily/weekly/monthly mail from cron and manually checking things.  All critical (and some non-critical) things are now being checked every 60 seconds, and I only have my attention stolen when there is an actionable issue found by mcrover.

So… an ode to the philosophy of an old system.  Don’t make me plow through a bunch of data to find the things I need to address.  I’ll do that when there’s a problem, not when there isn’t a problem.  For 7×24 general purpose computing devices running Linux, macOS or FreeBSD, I install and run the mcrover service and connect it to the mesh.  And it requires very little oomph; it runs just fine on a Raspberry Pi 3 or 4.

So why the weather display?  It’s just useful to me, particularly in the mowing season where I need to plan ahead for yard work.  And I’ve just grown tired of the weather websites.  Most are loaded with ads and clutter.  All of them are tracking us.  Why not just pull the data from tax-funded sources in JSON form and do it myself?  I’ve got a dedicated display which doesn’t have any alerts to display most of the time, so it made sense to put it there.

The Qt client using X11, showing the weather forecast.

mcrover Qt client using X11, showing the weather forecast

The Qt client using EGLFS, showing the weather radar.

The curses client, showing ASCII art since there are no alerts to be shown.

mcrover curses client with no alerts.
Posted on

Replaced IronWolf Pro 8TB with Ultrastar DC HC510 10TB

Due to a firmware problem in the Seagate IronWolf Pro 8TB drives that makes them incompatible with ZFS on FreeBSD, I returned them over the weekend and ordered a pair of Ultrastar DC HC510 10TB drives. I’ve had phenomenal results from Ultrastars in the past, and as near Its I can tell they’ve always been very good enterprise-grade drives regardless of the owner (IBM, Hitach, HGST, Western Digital). The Ultrastars arrived today, and I put them in the zfs1 pool:

# zpool list -v
NAME               SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zfs1              16.3T  2.13T  14.2T        -         -    10%    13%  1.00x  ONLINE  -
  mirror          3.62T  1.53T  2.09T        -         -    29%    42%
    gpt/gpzfs1_0      -      -      -        -         -      -      -
    gpt/gpzfs1_1      -      -      -        -         -      -      -
  mirror          3.62T   609G  3.03T        -         -    19%    16%
    gpt/gpzfs1_2      -      -      -        -         -      -      -
    gpt/gpzfs1_3      -      -      -        -         -      -      -
  mirror          9.06T  1.32M  9.06T        -         -     0%     0%
    gpt/gpzfs1_4      -      -      -        -         -      -      -
    gpt/gpzfs1_5      -      -      -        -         -      -      -

Everything seems good. Note that the scrub repair of 33.8G was due to me pulling the IronWolf drives from the chassis with the system live (after having removed them from the pool). This apparently caused a burp on the backplane, which was fully corrected by the scrub.

# zpool status
  pool: zfs1
 state: ONLINE
  scan: scrub repaired 33.8G in 0 days 04:43:10 with 0 errors on Sun Nov 10 01:45:59 2019
remove: Removal of vdev 2 copied 36.7G in 0h3m, completed on Thu Nov  7 21:26:09 2019
    111K memory used for removed device mappings
config:

	NAME              STATE     READ WRITE CKSUM
	zfs1              ONLINE       0     0     0
	  mirror-0        ONLINE       0     0     0
	    gpt/gpzfs1_0  ONLINE       0     0     0
	    gpt/gpzfs1_1  ONLINE       0     0     0
	  mirror-1        ONLINE       0     0     0
	    gpt/gpzfs1_2  ONLINE       0     0     0
	    gpt/gpzfs1_3  ONLINE       0     0     0
	  mirror-3        ONLINE       0     0     0
	    gpt/gpzfs1_4  ONLINE       0     0     0
	    gpt/gpzfs1_5  ONLINE       0     0     0

errors: No known data errors
Posted on

Expanded zfs1 pool on kiva

I purchased two Seagate IronWolf Pro 8TB drives at MicroCenter today. They’ve been added to the zfs1 pool on kiva.


# gpart create -s gpt da5
da5 created
# gpart create -s gpt da6
da6 created

# gpart add -t freebsd-zfs -l gpzfs1_4 -b1M -s7450G da5
da5p1 added
# gpart add -t freebsd-zfs -l gpzfs1_5 -b1M -s7450G da6
da6p1 added

# zpool add zfs1 mirror /dev/gpt/gpzfs1_4 /dev/gpt/gpzfs1_5

# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zfs1 14.5T 2.76T 11.7T - - 14% 19% 1.00x ONLINE -
mirror 3.62T 1.87T 1.75T - - 33% 51%
gpt/gpzfs1_0 - - - - - - -
gpt/gpzfs1_1 - - - - - - -
mirror 3.62T 910G 2.74T - - 24% 24%
gpt/gpzfs1_2 - - - - - - -
gpt/gpzfs1_3 - - - - - - -
mirror 7.25T 1.05M 7.25T - - 0% 0%
gpt/gpzfs1_4 - - - - - - -
gpt/gpzfs1_5 - - - - - - -

Posted on

mcperf: a multithreaded bandwidth tester

I’ve been really dismayed by the lack of decent simple tools for testing the available bandwidth between a pair of hosts above 1 gigabit/second. Back when I didn’t have any 10 gigabit connections at home, I used iperf and iperf3. But I now have several 10 gigabit connections on my home network, and since these tools don’t use multithreading effectively, they become CPU bound (on a single core) before they reach the target bandwidth. Tools like ssh and scp have the same problem; they’re single threaded and become CPU bound long before they saturate a 10 gigabit connection.

When I install a 10 gigabit connection, whether it’s via SFP+ DACs, SFP+ SR optics or 10GbaseT, it’s important that I’m able to test the connection’s ability to sustain somewhere near line rate transfers end-to-end. Especially when I’m buying my DACs, transceivers or shielded cat6a patch cables from eBay or any truly inexpensive vendor. I needed a tool that could saturate a 10 gigabit connection and report the data transfer rate at the application level. Obviously due to the additional data for protocol headers and link encapsulation, this number will be lower than the link-level bandwidth, but it’s the number that ultimately matters for an application.

So, I quickly hacked together a multithreaded application to test my connections at home. It will spawn the requested number of threads (on each end) and the server will send data from each thread. Each thread gets its own TCP connection.

For a quick hack, it works well.


dwm@www:/home/dwm% mcperf -t 4 -c kiva
bandwidth: 8.531 Gbits/sec
bandwidth: 8.922 Gbits/sec
bandwidth: 9.069 Gbits/sec
bandwidth: 9.148 Gbits/sec
bandwidth: 9.197 Gbits/sec
bandwidth: 9.230 Gbits/sec
bandwidth: 9.253 Gbits/sec
bandwidth: 9.269 Gbits/sec
bandwidth: 9.283 Gbits/sec

Given that I don’t create servers that don’t use strong authentication, even if they’ll only be run for 10 seconds, I’m using the PeerAuthenticator from libDwmAuth for authentication. No encryption of the data that’s being sent, since it’s not necessary.

Of course this got me thinking about the number of tools we have today that just don’t cut it in a 10 gigabit network. ssh, scp, ftp, fetch, etc. Even NFS code has trouble saturating a 10 gigabit connection. It seems like eons ago that Herb Sutter wrote “The Free Lunch Is Over”. It was published in 2005. Yet we still have a bunch of tools that are CPU bound due to being single-threaded. How are we supposed to take full advantage of 10 gigabit and faster networks if the tools we use for file transfer, streaming, etc. are single-threaded and hence CPU bound well before they reach 10 gigabits/second? What happens when I run some fiber at home for NAS and want to run 40 gigabit or (egads!) 100 gigabit? It’s not as if I don’t have the CPU to do 40 gigabits/second; my NAS has 12 cores and 24 threads. But if an application is single-threaded, it becomes CPU bound at around 3.5 gigabits/second on a typical server CPU core. 🙁 Sure, that’s better than 1 gigabit/second but it’s less than what a single SATA SSD can do, and much less than what an NVME/M.2/striped SATA SSD/et. al. can do.

We need tools that aren’t written as if it’s 1999. I suspect that after I polish up mcperf a little bit, I’m going to work on my own replacement for scp so I can at least transfer files without being CPU bound at well below my network bandwidth.

Posted on

short flurry of ssh login attempts blocked by mcblockd

mcblockd added quite a few networks during a 20 minute period today. I don’t have an explanation for the ssh login attempts all coming in during this period, but it’s nice to see that mcblockd happily blocked all of them.

While this is by no means a high rate of attempts, it’s higher than what I normally see.

May 22 11:32:10 ria mcblockd: [I] Added 185.129.60/22 (DK) to ssh_losers for 180 days
May 22 11:32:11 ria mcblockd: [I] Added 89.234.152/21 (FR) to ssh_losers for 180 days
May 22 11:32:45 ria mcblockd: [I] Added 46.233.0/18 (BG) to ssh_losers for 180 days
May 22 11:33:00 ria mcblockd: [I] Added 216.218.222/24 (US) to ssh_losers for 30 days
May 22 11:33:05 ria mcblockd: [I] Added 199.87.154/24 (CA) to ssh_losers for 30 days
May 22 11:33:15 ria mcblockd: [I] Added 78.109.16/20 (UA) to ssh_losers for 180 days
May 22 11:33:18 ria mcblockd: [I] Added 89.38.148/22 (FR) to ssh_losers for 180 days
May 22 11:33:26 ria mcblockd: [I] Added 65.19.167/24 (US) to ssh_losers for 30 days
May 22 11:34:05 ria mcblockd: [I] Added 62.212.64/19 (NL) to ssh_losers for 180 days
May 22 11:35:54 ria mcblockd: [I] Added 190.10.0/17 (CR) to ssh_losers for 180 days
May 22 11:37:16 ria mcblockd: [I] Added 192.42.116/22 (NL) to ssh_losers for 180 days
May 22 11:38:33 ria mcblockd: [I] Added 199.249.223/24 (US) to ssh_losers for 30 days
May 22 11:38:37 ria mcblockd: [I] Added 173.254.216/24 (US) to ssh_losers for 30 days
May 22 11:39:48 ria mcblockd: [I] Added 128.52.128/24 (US) to ssh_losers for 30 days
May 22 11:39:51 ria mcblockd: [I] Added 64.113.32/24 (US) to ssh_losers for 30 days
May 22 11:40:32 ria mcblockd: [I] Added 23.92.27/24 (US) to ssh_losers for 30 days
May 22 11:40:50 ria mcblockd: [I] Added 162.221.202/24 (CA) to ssh_losers for 30 days
May 22 11:42:42 ria mcblockd: [I] Added 91.213.8/24 (UA) to ssh_losers for 180 days
May 22 11:43:37 ria mcblockd: [I] Added 162.247.72/24 (US) to ssh_losers for 30 days
May 22 11:44:34 ria mcblockd: [I] Added 193.110.157/24 (NL) to ssh_losers for 180 days
May 22 11:44:38 ria mcblockd: [I] Added 128.127.104/23 (SE) to ssh_losers for 180 days
May 22 11:45:50 ria mcblockd: [I] Added 179.43.128/18 (CH) to ssh_losers for 180 days
May 22 11:45:55 ria mcblockd: [I] Added 89.144.0/18 (DE) to ssh_losers for 180 days
May 22 11:46:29 ria mcblockd: [I] Added 197.231.220/22 (LR) to ssh_losers for 180 days
May 22 11:46:44 ria mcblockd: [I] Added 195.254.132/22 (RO) to ssh_losers for 180 days
May 22 11:46:54 ria mcblockd: [I] Added 154.16.244/24 (US) to ssh_losers for 30 days
May 22 11:47:52 ria mcblockd: [I] Added 87.118.64/18 (DE) to ssh_losers for 180 days
May 22 11:48:51 ria mcblockd: [I] Added 46.165.224/19 (DE) to ssh_losers for 180 days
May 22 11:50:13 ria mcblockd: [I] Added 178.17.168/21 (MD) to ssh_losers for 180 days
May 22 11:50:47 ria mcblockd: [I] Added 31.41.216/21 (UA) to ssh_losers for 180 days
May 22 11:50:55 ria mcblockd: [I] Added 62.102.144/21 (SE) to ssh_losers for 180 days
May 22 11:51:19 ria mcblockd: [I] Added 64.137.244/24 (CA) to ssh_losers for 30 days
May 22 11:52:28 ria mcblockd: [I] Added 80.244.80/20 (SE) to ssh_losers for 180 days
May 22 11:52:42 ria mcblockd: [I] Added 192.160.102/24 (CA) to ssh_losers for 30 days
May 22 11:53:06 ria mcblockd: [I] Added 176.10.96/19 (CH) to ssh_losers for 180 days
May 22 11:55:38 ria mcblockd: [I] Added 77.248/14 (NL) to ssh_losers for 180 days
May 22 11:56:20 ria mcblockd: [I] Added 199.119.112/24 (US) to ssh_losers for 30 days
May 22 11:56:32 ria mcblockd: [I] Added 94.142.240/21 (NL) to ssh_losers for 180 days
Posted on

China is a lousy netizen

There’s no one even close in terms of ssh login attempts. In a span of two weeks, mcblockd has blocked 47 million more addresses from China. That doesn’t mean I’ve seen 47 million IP addresses in login attempts. It means that China has a lot of address space being used to probe U.S. sites.

Brazil is in second place, but they’re behind by more than a decimal order of magnitude. Below are the current top two countries being blocked by mcblockd, by quantity of address space.

% mcblockc getactive ssh_losers

...

  Addresses covered per country:
    CN 149,911,680
      /10 networks:   10 (41,943,040 addresses)
      /11 networks:   21 (44,040,192 addresses)
      /12 networks:   38 (39,845,888 addresses)
      /13 networks:   26 (13,631,488 addresses)
      /14 networks:   23 (6,029,312 addresses)
      /15 networks:   26 (3,407,872 addresses)
      /16 networks:   14 (917,504 addresses)
      /17 networks:    4 (131,072 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    2 (4,096 addresses)
      /22 networks:    2 (2,048 addresses)
      /25 networks:    1 (128 addresses)
    BR 14,170,112
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    3 (6,291,456 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    3 (1,572,864 addresses)
      /14 networks:    3 (786,432 addresses)
      /15 networks:    1 (131,072 addresses)
      /17 networks:    2 (65,536 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    5 (40,960 addresses)
      /20 networks:    2 (8,192 addresses)
      /21 networks:    5 (10,240 addresses)
      /22 networks:    4 (4,096 addresses)

I seriously doubt that Chinese citizens have anything to do with these attempts. I’m told that the Great Firewall blocks most ssh traffic on port 22. Not to mention that China’s Internet connectivity is somewhere near 95th in the world in terms of available bandwidth, so it’d be terribly painful for an ordinary user to use ssh or scp from China to my gateway. I think I can assume this is all government-sponsored probing.

Posted on

mcblockd has been busy

The mcblockd automation has been running for roughly one week. It’s been fairly busy automatically blocking those trying to crack my ssh server. Below is some of the output from a query of the active blocked networks (the summary information for the top 10 countries by the number of addresses being blocked). Interesting to note that the automation has blocked a huge swath of addresses from China. State-sponsored cyberattacks?

% mcblockc getactive ssh_losers

...

  Addresses covered per country:
    CN 102,263,808
      /10 networks:    8 (33,554,432 addresses)
      /11 networks:   17 (35,651,584 addresses)
      /12 networks:   21 (22,020,096 addresses)
      /13 networks:   11 (5,767,168 addresses)
      /14 networks:   14 (3,670,016 addresses)
      /15 networks:    9 (1,179,648 addresses)
      /16 networks:    6 (393,216 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    1 (2,048 addresses)
      /22 networks:    1 (1,024 addresses)
    KR 7,864,320
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    1 (2,097,152 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    1 (524,288 addresses)
    IN 7,340,032
      /10 networks:    1 (4,194,304 addresses)
      /12 networks:    2 (2,097,152 addresses)
      /13 networks:    1 (524,288 addresses)
      /14 networks:    2 (524,288 addresses)
    BR 7,252,992
      /11 networks:    3 (6,291,456 addresses)
      /13 networks:    1 (524,288 addresses)
      /14 networks:    1 (262,144 addresses)
      /15 networks:    1 (131,072 addresses)
      /17 networks:    1 (32,768 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    1 (2,048 addresses)
      /22 networks:    1 (1,024 addresses)
    FR 6,782,976
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    1 (2,097,152 addresses)
      /15 networks:    1 (131,072 addresses)
      /16 networks:    5 (327,680 addresses)
      /18 networks:    2 (32,768 addresses)
    AR 4,524,032
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    2 (1,048,576 addresses)
      /14 networks:    8 (2,097,152 addresses)
      /15 networks:    2 (262,144 addresses)
      /16 networks:    1 (65,536 addresses)
      /21 networks:    1 (2,048 addresses)
    JP 4,227,072
      /10 networks:    1 (4,194,304 addresses)
      /17 networks:    1 (32,768 addresses)
    RU 3,484,672
      /13 networks:    2 (1,048,576 addresses)
      /14 networks:    5 (1,310,720 addresses)
      /15 networks:    6 (786,432 addresses)
      /16 networks:    2 (131,072 addresses)
      /17 networks:    4 (131,072 addresses)
      /18 networks:    2 (32,768 addresses)
      /19 networks:    5 (40,960 addresses)
      /22 networks:    3 (3,072 addresses)
    IT 3,280,896
      /11 networks:    1 (2,097,152 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /15 networks:    1 (131,072 addresses)
      /20 networks:    1 (4,096 addresses)
    TW 2,637,824
      /12 networks:    2 (2,097,152 addresses)
      /13 networks:    1 (524,288 addresses)
      /18 networks:    1 (16,384 addresses)

...
Posted on

Looking at ‘Synners’ (TCP SYN data)

One of the many sets of data I collect with mcflow on my gateway is traffic counters for TCP SYN packets I receive but do not SYN ACK. I keep the source IP address, the destination port, and of course timestamps and counters. This type of data generally represents one of three things: probing for vulnerable services which I don’t run, probing for services I do run but block from offenders, or probing for botnet-controlled devices.

The table below shows the top 10 ports for the current week. In the case of ssh and http, I do run those services but mcblockd automatically blocks those who violate my configured policies. I do not run a telnet server anywhere (my IoT devices are of my own design and use ECDH, 2048-bit RSA keys and AES128). I also do not run MS SQL Server or rdp (Remote Desktop). I have no Windows hosts, and if I did, I certainly wouldn’t expose MS SQL Server or Remote Desktop.

Ports 7547 and 5358 are known to be used by Mirai and its descendants. Port 7547 is also a common port used by broadband ISPs for TR-064 services (specifically, TR-069) to manage home routers.

Port Packets Bytes
22 (ssh) 22116 1168688
23 (telnet) 3740 152784
80 (http) 1601 99216
1433 (ms-sql-s) 1279 52288
81 917 38016
7547 515 20620
3389 (rdp) 199 8792
5358 195 8148
2323 181 7384
8080 154 6700

Below is a table showing the SYNs I didn’t SYN ACK by country. This is just the top 10. Note that the top two have large swaths of their IP address space automatically blocked by mcblockd for violating my configured policies. They’re also known state sponsors of cyberattacks, and the evidence is pretty clear here. Much (but not all) of the US stuff is research scanning.

Country Packets Bytes
RU (Russian Federation) 17394 864024
CN (China) 6038 319116
US (United States) 3077 169932
NL (Netherlands) 1160 47580
TH (Thailand) 603 33480
UA (Ukraine) 467 20612
KR (Korea) 462 19380
BR (Brazil) 426 18708
FR (France) 341 17828
TR (Turkey) 281 11756

What is perhaps interesting about this data: the lines drawn during WWII and the Cold War don’t appear to have changed. I find this very sad. I’m just a tiny single user running a very modest home network, yet I’m a target of Russia and China. And my network is likely much more secure than the average home network. I assume this means that all of us are being probed all of the time, and some of us are probably regularly compromised. I think we (meaning the entire industry) need to consider completely banning telnet and doing something real about securing IoT devices.