Expanded zfs1 pool on kiva

I purchased two Seagate IronWolf Pro 8TB drives at MicroCenter today. They’ve been added to the zfs1 pool on kiva.

# gpart create -s gpt da5
da5 created
# gpart create -s gpt da6
da6 created

# gpart add -t freebsd-zfs -l gpzfs1_4 -b1M -s7450G da5
da5p1 added
# gpart add -t freebsd-zfs -l gpzfs1_5 -b1M -s7450G da6
da6p1 added

# zpool add zfs1 mirror /dev/gpt/gpzfs1_4 /dev/gpt/gpzfs1_5

# zpool list -v
zfs1 14.5T 2.76T 11.7T - - 14% 19% 1.00x ONLINE -
mirror 3.62T 1.87T 1.75T - - 33% 51%
gpt/gpzfs1_0 - - - - - - -
gpt/gpzfs1_1 - - - - - - -
mirror 3.62T 910G 2.74T - - 24% 24%
gpt/gpzfs1_2 - - - - - - -
gpt/gpzfs1_3 - - - - - - -
mirror 7.25T 1.05M 7.25T - - 0% 0%
gpt/gpzfs1_4 - - - - - - -
gpt/gpzfs1_5 - - - - - - -

AMZL US sucks, Amazon no longer my preferred online store

It appears that Amazon’s greed is continuing to degrade the customer experience.

Many areas of the US have had AMZL US become the primary logistics and delivery service for Amazon Prime shipments over the last couple of years. The problem is that for many of us, it has largely eliminated the incentive for Prime membership: fast, ‘free’ shipping. Of course it’s NOT free, since Prime membership is not free. But the bigger problem is that for many of us, AMZL US is dreadfully bad compared to USPS, UPS, FedEx or DHL.

I’d estimate that most of the Prime orders delivered to my home via AMZL US have not met the Prime promise. They’re often not on time, or even close to on time. 2 days becomes 4 to 7 days. On many occasions the final tracking record says “handed to resident” when in fact that was not true (no one was home, package was left in driveway). And recently, a package arrived with the outer box intact, and the product’s box (inside the outer box) intact but EMPTY. And all of my recent AMZL deliveries have been late by at least a day. Today’s notice is typical of what I’ve seen lately:

Note that ‘Wednesday’ is 5 days after I ordered. This is a small Prime item (would easily fit in my mailbox and hence USPS would be inexpensive), as was the item that didn’t arrive (the empty box shipment). And these are just a couple of the recent issues. Less than 20 percent of my AMZL shipments have been completely logistically correct. All of my recent shipments have found their way into the abyss above; delayed on the day they were supposed to arrive, at which point they can’t tell you when it will arrive or when they’ll even ATTEMPT to deliver it. This isn’t how FedEx, UPS or even the USPS do things. They have actual logistics, while AMZL apparently does not. AMZL can not reliably deliver packages on time, nor reliably track them. And of course, the day they expected to deliver it was a Sunday. Umm, I don’t need or even want Sunday deliveries. Especially if it triggers the “we no longer know when we’ll deliver your package” tracking status when that Sunday passes.

This is what happens when a company decides it would like to leverage its increasingly monopolistic position to make higher margins. As near as I can tell from stories from AMZL drivers, former AMZL logistics employees, other customers and my own experiences, AMZL is a logistics morass. And the last mile, arguably the most critical, is essentially slave labor. As the old adage goes, you get what you pay for. This isn’t open capitalist markets choosing the winner; as near as I can tell, there are almost no customers who prefer AMZL over USPS, UPS, FedEx, DHL, etc. And there are many stories of customers dropping their Prime membership because they can’t control who is used as a courier and get stuck with AMZL. This is Amazon deciding that they’d like to squeeze a few more pennies from their business by underpaying for courier services. Who suffers initially? Amazon Prime customers, and those who think they might build a profitable business delivering products sold by Amazon (good luck; the last time I ran the numbers, it was worse than trying to make a living as an Uber driver). Who suffers in the long run? Amazon and its shareholders. When Walmart and BestBuy start looking like significantly better options to your customers, you know you’re in the running in the race to the bottom.

This isn’t a bottom-up problem. While I’m sure there are some bad apples in the driver and contract courier company ranks, the real cause is much more likely the pricing demanded by Amazon. This is an Amazon initiative, and from my narrow view, very poorly implemented. I’m quickly becoming an alienated customer, and it’s been made clear that they don’t really give a rat’s ass about it since I can’t blacklist their AMZL service. Prime is now mostly a contract that’s regularly broken by one party (Amazon).

It’s unlikely that I’ll renew my Prime membership when it comes due. Nearly everything I buy from Amazon is available elsewhere, with free shipping from a RELIABLE courier service, and often at a lower price. Since 2005 I’ve been preferring Amazon because my Prime membership yielded fast, reliable delivery (via UPS or FedEx). That’s no longer true, and I have no recourse other than ditching my Prime membership and shopping elsewhere. Amazon doesn’t allow me to blacklist their lousy AMZL delivery service. The time alone that I’ve spent chasing down AMZL delivery issues costs more than the annual Prime subscription.

Today I spent $240 at my local BestBuy on an item I had intended to buy from Amazon (exact same price). And I generally hate BestBuy. But… walking out of a brick and mortar with product in hand is orders of magnitude better than waiting 5 to 7 days for something to POSSIBLY arrive and paying extra (Prime membership fee) for that ‘privilege’.

Who will get more of my business? For technology, the usual suspects: NewEgg, B&H, Adorama, Microcenter and BestBuy. For tools, Home Depot, Lowe’s, Menard’s, Performance Line Tool Center, Tooltopia and others.

To be clear, I’m not oblivious to the problems of scale with respect to Amazon delivery. And I’m _far_ from a luddite; I strongly believe in technological advancement and don’t need a human to hand me a package. But AMZL has been around since 2014 and it still sucks for many of us. I didn’t sign up for this experiment; I signed up (and paid for) 2-day delivery. If you ask me, a smarter move on Amazon’s part would have been to use AMZL as the free (as in Prime membership not required) delivery service, and kept the reliable courier services as the only ones used for Prime membership. And been willing to invest more in making AMZL viable before forcing it on customers who are paying extra for 2-day delivery service.

You may need 10 gigabit networking at home

Given the ignorance I’ve seen in some forums with respect to the need for 10 gigabit networking at home, I decided it was time to blog about it.

The argument against 10 gigabit networking at home for ANYONE, as I’ve seen posted in discussions on arstechnica and other sites, is ignorant. The same arguments were made when we got 100baseT, and again when we got 1000baseT at consumer pricing. And they were proven wrong, just as they will be for 10G whether it’s 10GBaseT, SFP+ DAC, SFP+ SR optics, or something else.

First, we should disassociate the WAN connection (say Comcast, Cox, Verizon, whatever) from the LAN. If you firmly believe that you don’t need LAN speeds that are higher than your WAN connection, I have to assume that you either a) do very little within your home that doesn’t use your WAN connection or b) just have no idea what the heck you’re talking about. If you’re in the first camp, you don’t need 10 gigabit LAN in your home. If you’re in the second camp, I can only encourage you to learn a bit more and use logic to determine your own needs. And stop telling others what they don’t need without listening to their unique requirements.

There are many of us with needs for 10 gigabit LAN at home. Let’s take my needs, for example, which I consider modest. I have two NAS boxes with ZFS arrays. One of these hosts some automated nightly backups, a few hundred movies (served via Plex) and some of my music collection. The second hosts additional automated nightly backups, TimeMachine instances and my source code repository (which is mirrored to the first NAS with ZFS incremental snapshots).

At the moment I have 7 machines that run automated backups over my LAN. I consider these backups critical to my sanity, and they’re well beyond what I can reasonably accomplish via a cloud storage service. With data caps and my outbound bandwidth constraints, nightly cloud backups aren’t an option. Fortunately, I am not in desperate need of offsite backups, and the truly critical stuff (like my source code repository) is mirrored in a lot of places and occasionally copied to DVD for offsite storage. I’m not sure what I’ll do the day my source code repository gets beyond what I can reasonably burn to DVD but it’ll be a long while before I get there (if ever). If I were to have a fire, I’d only need to grab my laptop on my way out the door in order to save my source code. Yes, I’d lose other things. But…

Fires are rare. I hope to never have one. Disk failures are a lot less rare. As are power supply failures, fan failures, etc. This is the main reason I use ZFS. But, at 1 gigabit/second network speeds, the network is a bottleneck for even a lowly single 7200 rpm spinning drive doing sequential reads. A typical decent single SATA SSD will fairly easily reach 4 gigabits/second. Ditto for a small array of spinning drives. NVME/M.2/multiple SSD/larger spinning drive array/etc. can easily go beyond 10 gigabits/second.

Why does this matter? When a backup kicks off and saturates a 1 gigabit/second network connection, that connection becomes a lot less usable for other things. I’d prefer the network connection not be saturated, and that the backup complete as quickly as possible. In other words, I want to be I/O bound in the storage subsystem, not bandwidth bound in the network. This becomes especially critical when I need to restore from a backup. Even if I have multiple instances of a service (which I do in some cases), there’s always one I consider ‘primary’ and want to restore as soon as possible. And if I’m restoring from backup due to a security breach (hasn’t happened in 10 years, knock on wood), I probably can’t trust any of my current instances and hence need a restore from backup RIGHT NOW, not hours later. The faster a restoration can occur (even if it’s just spinning up a VM snapshot), the sooner I can get back to doing real work.

Then there’s just the shuffling of data. Once in a while I mirror all of my movie files, just so I don’t have to re-rip a DVD or Blu-Ray. Some of those files are large, and a collection of them is very large. But I have solid state storage in all of my machines and arrays of spinning drives in my NAS machines. Should be fast to transfer the files, right? Not if your network is 1 gigabit/second… your average SATA SSD will be 75% idle while trying to push data through a 1 gigabit/second network, and NVMe/M.2/PCIe solid state will likely be more than 90% idle. In other words, wasting time. And time is money.

So, some of us (many of us if you read servethehome.com) need 10 gigabit networking at home. And it’s not ludicrously expensive anymore, and prices will continue to drop. While I got an exceptional deal on my Netgear S3300-52X-POE+ main switch ($300), I don’t consider it a major exception. New 8-port 10GbaseT switches are here for under $1000, and SFP+ switches are half that price new (say a Ubiquiti ES-16-XG which also has 4 10GbaseT ports). Or buy a Quanta LB6M for $300 and run SR optics. Right now I have a pair of Mellanox ConnectX-2 EN cards for my web server and my main NAS, which I got for $17 each. Two 3-meter DAC cables for $15 each from fs.com connect these to the S3300-52X-POE+ switch’s SFP+ ports. In my hackintosh desktop I have an Intel X540-T2 card, which is connected to one of the 10GbaseT ports on the S3300-52X-POE+ via cat6a shielded keystones and cables (yes, my patch panels are properly grounded). I will eventually change the X540-T2 to a less power-hungry card, but it works for now and it was $100. I expect to see more 10GbaseT price drops in 2018. And I hope to see more options for mixed SFP+ and 10GbaseT in switches. We’re already at the point where copper has become unwieldy, since cat6a (esp. shielded) and cat7 are thick, heavy cables. And cat8? Forget about running much of that since it’s a monster size-wise. At 10 gigabits/second, it already makes sense to run multimode fiber for EMI immunity, distance, raceway/conduit space, no code violations when co-resident with AC power feeds, etc. Beyond 10 gigabit/second, which we’ll eventually want and need, I don’t see copper as viable. Sure, copper has traditionally been easier to terminate than fiber. But in part that’s because the consumer couldn’t afford or justify the need for it and hence fiber was a non-consumer technology. Today it’s easier to terminate fiber than it’s ever been, and it gets easier all the time. And once you’re pulling a cat6a or cat8 cable, you can almost fit an OM4 fiber cable with a dual LC connector on it through the same spaces and not have to field terminate at all. That’s the issue we’re facing with copper. Much like the issues with CPU clock speeds, we’re reaching the limits of what can reasonably be run on copper over typical distances in a home (where cable routes are often far from the shortest path from A to B). In a rack, SFP+ DAC (Direct Attach Copper) cables work well. But once you leave the rack and need to go through a few walls, the future is fiber. And it’ll arrive faster than some people expect, in our homes. Just check what it takes to send 4K raw video at 60fps. Or to backhaul an 802.11ac Wave 2 WiFi access point without creating a bottleneck on the wired network. Or the time required to send that 4TB full backup to your NAS.

OK, I feel better. 🙂 I had to post about this because it’s just not true that no one needs 10 gigabit networking in their home. Some people do need it.

My time is valuable, as is yours. Make your own decisions about what makes sense for your own home network based on your own needs. If you don’t have any pain points with your existing network, keep it! Networking technology is always cheaper next year than it is today. But if you can identify pain that’s caused by bandwidth constraints on your 1 gigabit network, and the pain warrants an upgrade to 10 gigabit (even if only between 2 machines), by all means go for it! I don’t know anyone that’s ever regretted a network upgrade that was well considered.

Note that this post came about partly due to some utter silliness I’ve seen posted online, including egregiously incorrect arithmetic. One of my favorites was from a poster on arstechnica who repeatedly (as in dozens of times) claimed that no one needed 10 gigabit ethernet at home because he could copy a 10 TB NAS to another NAS in 4 hours over a 1 gigabit connection. So be careful what you read on the Internet, especially if it involves numbers… it might be coming from someone with faulty arithmetic that certainly hasn’t ever actually copied 10 terabytes of data over a 1 gigabit network in 4 hours (hint… it would take almost 24 hours if it has the network all to itself, longer if there is other traffic on the link).

I’d be remiss if I didn’t mention other uses for 10 gigabit ethernet. Does your household do a lot of gaming via Steam? You’d probably benefit from having a local Steam cache with 10 gigabit connectivity to the gaming machines. Are you running a bunch of Windows 10 instances? You can pull down updates to one machine and distribute them from there to all of your Windows 10 instances, and the faster, the better. Pretty much every scenario where you need to move large pieces of data over the network will benefit from 10 gigabit ethernet. You have to decide for yourself if the cost is justified. In my case, I’ve installed the bare minimum (4 ports of 10 gigabit) that alleviates my existing pain points. At some point in the future I’ll need more 10 gigabit ports, and as long as it’s not in the next few months, it’ll be less expensive than it is today. But if you could use it today, take a look at your inexpensive options. Mellanox ConnectX-2 EN cards are inexpensive on eBay, and even the newer cards aren’t ludicrously expensive. If you only need 3 meters or less of distance, look at using SFP+ DAC cables. If you need more distance, look at using SR optical transceivers in Mellanox cards or Intel X540-DA2 (or newer) and fiber, or 10GbaseT (Intel X540-T2 or X540-T1 or newer, or a motherboard with on-board 10GbaseT). You have relatively inexpensive switch options if you’re willing to buy used on eBay and only need a few ports at 10 gigabit, or you’re a techie willing to learn to use a Quanta LB6M and can put it somewhere where it won’t drive you crazy (it’s loud).

mcperf: a multithreaded bandwidth tester

I’ve been really dismayed by the lack of decent simple tools for testing the available bandwidth between a pair of hosts above 1 gigabit/second. Back when I didn’t have any 10 gigabit connections at home, I used iperf and iperf3. But I now have several 10 gigabit connections on my home network, and since these tools don’t use multithreading effectively, they become CPU bound (on a single core) before they reach the target bandwidth. Tools like ssh and scp have the same problem; they’re single threaded and become CPU bound long before they saturate a 10 gigabit connection.

When I install a 10 gigabit connection, whether it’s via SFP+ DACs, SFP+ SR optics or 10GbaseT, it’s important that I’m able to test the connection’s ability to sustain somewhere near line rate transfers end-to-end. Especially when I’m buying my DACs, transceivers or shielded cat6a patch cables from eBay or any truly inexpensive vendor. I needed a tool that could saturate a 10 gigabit connection and report the data transfer rate at the application level. Obviously due to the additional data for protocol headers and link encapsulation, this number will be lower than the link-level bandwidth, but it’s the number that ultimately matters for an application.

So, I quickly hacked together a multithreaded application to test my connections at home. It will spawn the requested number of threads (on each end) and the server will send data from each thread. Each thread gets its own TCP connection.

For a quick hack, it works well.

dwm@www:/home/dwm% mcperf -t 4 -c kiva
bandwidth: 8.531 Gbits/sec
bandwidth: 8.922 Gbits/sec
bandwidth: 9.069 Gbits/sec
bandwidth: 9.148 Gbits/sec
bandwidth: 9.197 Gbits/sec
bandwidth: 9.230 Gbits/sec
bandwidth: 9.253 Gbits/sec
bandwidth: 9.269 Gbits/sec
bandwidth: 9.283 Gbits/sec

Given that I don’t create servers that don’t use strong authentication, even if they’ll only be run for 10 seconds, I’m using the PeerAuthenticator from libDwmAuth for authentication. No encryption of the data that’s being sent, since it’s not necessary.

Of course this got me thinking about the number of tools we have today that just don’t cut it in a 10 gigabit network. ssh, scp, ftp, fetch, etc. Even NFS code has trouble saturating a 10 gigabit connection. It seems like eons ago that Herb Sutter wrote “The Free Lunch Is Over”. It was published in 2005. Yet we still have a bunch of tools that are CPU bound due to being single-threaded. How are we supposed to take full advantage of 10 gigabit and faster networks if the tools we use for file transfer, streaming, etc. are single-threaded and hence CPU bound well before they reach 10 gigabits/second? What happens when I run some fiber at home for NAS and want to run 40 gigabit or (egads!) 100 gigabit? It’s not as if I don’t have the CPU to do 40 gigabits/second; my NAS has 12 cores and 24 threads. But if an application is single-threaded, it becomes CPU bound at around 3.5 gigabits/second on a typical server CPU core. 🙁 Sure, that’s better than 1 gigabit/second but it’s less than what a single SATA SSD can do, and much less than what an NVME/M.2/striped SATA SSD/et. al. can do.

We need tools that aren’t written as if it’s 1999. I suspect that after I polish up mcperf a little bit, I’m going to work on my own replacement for scp so I can at least transfer files without being CPU bound at well below my network bandwidth.

Dremel US40 Ultra Saw first impressions

On July 19, 2017 I bought a Dremel US40 Ultra Saw kit from Home Depot for $99 (marked down from $129). It came with 4 blades, the handle, the Allen wrench for blade changes and a carrying bag.

I bought it mainly for trimming off hardwood flooring at the wall after installation. I don’t normally need to do this, but occasionally I’ll cut a board a tad too long. Since my sliding compound miter saw is in the garage, it can be a hassle to run back and forth from the second floor (where I’m installing flooring right now) just to trim 1/8″ off of a board. It ruins my mojo.

I’d love to own a Festool TS55 or TS75, but the reality is that it’s a lot of money for a tool I don’t desperately need at the moment. I own a table saw, a sliding compound miter saw, a circular saw, a Bosch handheld jigsaw, a reciprocating saw (which I’m misplaced!), an oscillating tool, a 5″ variable speed grinder, a tile wetsaw, a metal chop saw, etc. A TS55 or TS75 is the king of accurate breakdown of sheet goods, but all I needed today was a means of trimming off hardwood flooring post-installation.

The Dremel US40 works well for this task. The main drawback is that it creates a tremendous amount of dust due to the blade design. The dust port adapter is nearly impossible to find in a store or online, and the reviews are not favorable because it falls out too easily. And it looks like an afterthought to me; the dust port on the saw is tiny and on the opposite side of the saw body from the blade, at a 90 degree angle. And when using the flush-cut blade, I suspect it does nothing at all since the flush cut blade is outside of the blade guard.

The second drawback: the blades are pricey and I suspect they don’t last very long. This has been one of my complaints about Dremel for a long time. They’re like the Gillette of tool companies: the tools aren’t terribly expensive, but they gouge you for blades, grinding wheels, etc.

But the tool itself is not bad. I think it’s more versatile than the Rockwell Versacut and similar, mainly because it can make an almost flush cut. I can’t speak to the longevity yet, but for $99 I’m not expecting the worm drive to last forever. But for special uses, I think it’s a good tool. For heavy, rough work I have better, more robust tools.

One thing that would make it a better tool: variable speed control. 17,000 rpm is fine for cutting wood, but I can imagine wanting a slower speed when cutting metal. Not to mention using the surface prep wheel.

A foot attachment that allowed miter cuts would also be nice, though I realize it would be limited since the depth of cut at 90 is already shallow at 3/4″.

The bag is definitely not something to write home about. No padding, no pockets, etc. I’ll wind up putting everything in a Systainer 2 at some point.

short flurry of ssh login attempts blocked by mcblockd

mcblockd added quite a few networks during a 20 minute period today. I don’t have an explanation for the ssh login attempts all coming in during this period, but it’s nice to see that mcblockd happily blocked all of them.

While this is by no means a high rate of attempts, it’s higher than what I normally see.

May 22 11:32:10 ria mcblockd: [I] Added 185.129.60/22 (DK) to ssh_losers for 180 days
May 22 11:32:11 ria mcblockd: [I] Added 89.234.152/21 (FR) to ssh_losers for 180 days
May 22 11:32:45 ria mcblockd: [I] Added 46.233.0/18 (BG) to ssh_losers for 180 days
May 22 11:33:00 ria mcblockd: [I] Added 216.218.222/24 (US) to ssh_losers for 30 days
May 22 11:33:05 ria mcblockd: [I] Added 199.87.154/24 (CA) to ssh_losers for 30 days
May 22 11:33:15 ria mcblockd: [I] Added 78.109.16/20 (UA) to ssh_losers for 180 days
May 22 11:33:18 ria mcblockd: [I] Added 89.38.148/22 (FR) to ssh_losers for 180 days
May 22 11:33:26 ria mcblockd: [I] Added 65.19.167/24 (US) to ssh_losers for 30 days
May 22 11:34:05 ria mcblockd: [I] Added 62.212.64/19 (NL) to ssh_losers for 180 days
May 22 11:35:54 ria mcblockd: [I] Added 190.10.0/17 (CR) to ssh_losers for 180 days
May 22 11:37:16 ria mcblockd: [I] Added 192.42.116/22 (NL) to ssh_losers for 180 days
May 22 11:38:33 ria mcblockd: [I] Added 199.249.223/24 (US) to ssh_losers for 30 days
May 22 11:38:37 ria mcblockd: [I] Added 173.254.216/24 (US) to ssh_losers for 30 days
May 22 11:39:48 ria mcblockd: [I] Added 128.52.128/24 (US) to ssh_losers for 30 days
May 22 11:39:51 ria mcblockd: [I] Added 64.113.32/24 (US) to ssh_losers for 30 days
May 22 11:40:32 ria mcblockd: [I] Added 23.92.27/24 (US) to ssh_losers for 30 days
May 22 11:40:50 ria mcblockd: [I] Added 162.221.202/24 (CA) to ssh_losers for 30 days
May 22 11:42:42 ria mcblockd: [I] Added 91.213.8/24 (UA) to ssh_losers for 180 days
May 22 11:43:37 ria mcblockd: [I] Added 162.247.72/24 (US) to ssh_losers for 30 days
May 22 11:44:34 ria mcblockd: [I] Added 193.110.157/24 (NL) to ssh_losers for 180 days
May 22 11:44:38 ria mcblockd: [I] Added 128.127.104/23 (SE) to ssh_losers for 180 days
May 22 11:45:50 ria mcblockd: [I] Added 179.43.128/18 (CH) to ssh_losers for 180 days
May 22 11:45:55 ria mcblockd: [I] Added 89.144.0/18 (DE) to ssh_losers for 180 days
May 22 11:46:29 ria mcblockd: [I] Added 197.231.220/22 (LR) to ssh_losers for 180 days
May 22 11:46:44 ria mcblockd: [I] Added 195.254.132/22 (RO) to ssh_losers for 180 days
May 22 11:46:54 ria mcblockd: [I] Added 154.16.244/24 (US) to ssh_losers for 30 days
May 22 11:47:52 ria mcblockd: [I] Added 87.118.64/18 (DE) to ssh_losers for 180 days
May 22 11:48:51 ria mcblockd: [I] Added 46.165.224/19 (DE) to ssh_losers for 180 days
May 22 11:50:13 ria mcblockd: [I] Added 178.17.168/21 (MD) to ssh_losers for 180 days
May 22 11:50:47 ria mcblockd: [I] Added 31.41.216/21 (UA) to ssh_losers for 180 days
May 22 11:50:55 ria mcblockd: [I] Added 62.102.144/21 (SE) to ssh_losers for 180 days
May 22 11:51:19 ria mcblockd: [I] Added 64.137.244/24 (CA) to ssh_losers for 30 days
May 22 11:52:28 ria mcblockd: [I] Added 80.244.80/20 (SE) to ssh_losers for 180 days
May 22 11:52:42 ria mcblockd: [I] Added 192.160.102/24 (CA) to ssh_losers for 30 days
May 22 11:53:06 ria mcblockd: [I] Added 176.10.96/19 (CH) to ssh_losers for 180 days
May 22 11:55:38 ria mcblockd: [I] Added 77.248/14 (NL) to ssh_losers for 180 days
May 22 11:56:20 ria mcblockd: [I] Added 199.119.112/24 (US) to ssh_losers for 30 days
May 22 11:56:32 ria mcblockd: [I] Added 94.142.240/21 (NL) to ssh_losers for 180 days

China is a lousy netizen

There’s no one even close in terms of ssh login attempts. In a span of two weeks, mcblockd has blocked 47 million more addresses from China. That doesn’t mean I’ve seen 47 million IP addresses in login attempts. It means that China has a lot of address space being used to probe U.S. sites.

Brazil is in second place, but they’re behind by more than a decimal order of magnitude. Below are the current top two countries being blocked by mcblockd, by quantity of address space.

% mcblockc getactive ssh_losers


  Addresses covered per country:
    CN 149,911,680
      /10 networks:   10 (41,943,040 addresses)
      /11 networks:   21 (44,040,192 addresses)
      /12 networks:   38 (39,845,888 addresses)
      /13 networks:   26 (13,631,488 addresses)
      /14 networks:   23 (6,029,312 addresses)
      /15 networks:   26 (3,407,872 addresses)
      /16 networks:   14 (917,504 addresses)
      /17 networks:    4 (131,072 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    2 (4,096 addresses)
      /22 networks:    2 (2,048 addresses)
      /25 networks:    1 (128 addresses)
    BR 14,170,112
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    3 (6,291,456 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    3 (1,572,864 addresses)
      /14 networks:    3 (786,432 addresses)
      /15 networks:    1 (131,072 addresses)
      /17 networks:    2 (65,536 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    5 (40,960 addresses)
      /20 networks:    2 (8,192 addresses)
      /21 networks:    5 (10,240 addresses)
      /22 networks:    4 (4,096 addresses)

I seriously doubt that Chinese citizens have anything to do with these attempts. I’m told that the Great Firewall blocks most ssh traffic on port 22. Not to mention that China’s Internet connectivity is somewhere near 95th in the world in terms of available bandwidth, so it’d be terribly painful for an ordinary user to use ssh or scp from China to my gateway. I think I can assume this is all government-sponsored probing.

mcblockd’s latest trick works: drop TCP connections

Evidence in the logs of mcblockd’s latest feature working. It’s successfully killing TCP connections when it adds a prefix to one of the pf tables.

Apr 29 03:42:40 ria mcblockd: [I] Dropped TCP connection from
Apr 29 03:42:40 ria mcblockd: [I] Added 221.144/12 (KR) to ssh_losers for 180 days
Apr 29 05:02:02 ria mcblockd: [I] Dropped TCP connection from
Apr 29 05:02:02 ria mcblockd: [I] Added 46.118/15 (UA) to ssh_losers for 180 days
Apr 29 07:07:42 ria mcblockd: [I] Dropped TCP connection from
Apr 29 07:07:42 ria mcblockd: [I] Added 120.128/13 (CN) to ssh_losers for 180 days
Apr 29 10:04:23 ria mcblockd: [I] Dropped TCP connection from
Apr 29 10:04:23 ria mcblockd: [I] Added 95.215.0/22 (RU) to ssh_losers for 180 days
Apr 29 11:51:34 ria mcblockd: [I] Dropped TCP connection from
Apr 29 11:51:34 ria mcblockd: [I] Added 110.240/12 (CN) to ssh_losers for 180 days
Apr 29 12:22:42 ria mcblockd: [I] Dropped TCP connection from
Apr 29 12:22:42 ria mcblockd: [I] Added 183.184/13 (CN) to ssh_losers for 180 days
Apr 29 13:13:54 ria mcblockd: [I] Dropped TCP connection from
Apr 29 13:13:54 ria mcblockd: [I] Dropped TCP connection from
Apr 29 13:13:54 ria mcblockd: [I] Added 120.144/12 (AU) to ssh_losers for 180 days
Apr 29 14:42:30 ria mcblockd: [I] Dropped TCP connection from
Apr 29 14:42:30 ria mcblockd: [I] Added 113.209/16 (CN) to ssh_losers for 180 days

mcblockd’s latest tricks: kill pf state, walk PCB list and kill TCP connections

Today I added a new feature to mcblockd to kill pf state for all hosts in a prefix when the prefix is added to one of my pf tables. This isn’t exactly what I want, but it’ll do for now.

mcblockd also now walks the PCB (protocol control block) list and drops TCP connections for hosts in a prefix I’ve just added to a pf table. Fortunately there was sample code in /usr/src/usr.sbin/tcpdrop/tcpdrop.c. The trick here is that I don’t currently have a means of mapping a pf table to where it’s applied (which ports, which interfaces). In the long term I might add code to figure that out, but in the interim I can configure ports and interfaces in mcblockd’s configuration file that will allow me to drop specific connections. For this first pass, I just toast all PCBs for a prefix.

The reason I added this feature: I occasionally see simultaneous login attempts from different IP addresses in the same prefix. If I’m going to block the prefix automatically, I want to cut off all of their connections right now, not after all of their connections have ended. Blowing away their pf state works, but leaves a hanging TCP connection in the ESTABLISHED state for a while. I want the PCBs to be cleaned up.

mcblockd has been busy

The mcblockd automation has been running for roughly one week. It’s been fairly busy automatically blocking those trying to crack my ssh server. Below is some of the output from a query of the active blocked networks (the summary information for the top 10 countries by the number of addresses being blocked). Interesting to note that the automation has blocked a huge swath of addresses from China. State-sponsored cyberattacks?

% mcblockc getactive ssh_losers


  Addresses covered per country:
    CN 102,263,808
      /10 networks:    8 (33,554,432 addresses)
      /11 networks:   17 (35,651,584 addresses)
      /12 networks:   21 (22,020,096 addresses)
      /13 networks:   11 (5,767,168 addresses)
      /14 networks:   14 (3,670,016 addresses)
      /15 networks:    9 (1,179,648 addresses)
      /16 networks:    6 (393,216 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    1 (2,048 addresses)
      /22 networks:    1 (1,024 addresses)
    KR 7,864,320
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    1 (2,097,152 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    1 (524,288 addresses)
    IN 7,340,032
      /10 networks:    1 (4,194,304 addresses)
      /12 networks:    2 (2,097,152 addresses)
      /13 networks:    1 (524,288 addresses)
      /14 networks:    2 (524,288 addresses)
    BR 7,252,992
      /11 networks:    3 (6,291,456 addresses)
      /13 networks:    1 (524,288 addresses)
      /14 networks:    1 (262,144 addresses)
      /15 networks:    1 (131,072 addresses)
      /17 networks:    1 (32,768 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    1 (2,048 addresses)
      /22 networks:    1 (1,024 addresses)
    FR 6,782,976
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    1 (2,097,152 addresses)
      /15 networks:    1 (131,072 addresses)
      /16 networks:    5 (327,680 addresses)
      /18 networks:    2 (32,768 addresses)
    AR 4,524,032
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    2 (1,048,576 addresses)
      /14 networks:    8 (2,097,152 addresses)
      /15 networks:    2 (262,144 addresses)
      /16 networks:    1 (65,536 addresses)
      /21 networks:    1 (2,048 addresses)
    JP 4,227,072
      /10 networks:    1 (4,194,304 addresses)
      /17 networks:    1 (32,768 addresses)
    RU 3,484,672
      /13 networks:    2 (1,048,576 addresses)
      /14 networks:    5 (1,310,720 addresses)
      /15 networks:    6 (786,432 addresses)
      /16 networks:    2 (131,072 addresses)
      /17 networks:    4 (131,072 addresses)
      /18 networks:    2 (32,768 addresses)
      /19 networks:    5 (40,960 addresses)
      /22 networks:    3 (3,072 addresses)
    IT 3,280,896
      /11 networks:    1 (2,097,152 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /15 networks:    1 (131,072 addresses)
      /20 networks:    1 (4,096 addresses)
    TW 2,637,824
      /12 networks:    2 (2,097,152 addresses)
      /13 networks:    1 (524,288 addresses)
      /18 networks:    1 (16,384 addresses)