Making my own office furniture: Part 3

On Sunday, I cut the base piece of plywood for the drawer cabinet that will also serve as a printer stand. I also installed half of the pocket hole plugs, trimmed them and sanded them flat. I have 8 more to do for the top of the cabinet.

On Monday, the 2″ casters arrived for the drawer cabinet that will also serve as a printer star. They’ll be installed in the base I cut on Sunday. I’m still waiting for the nut inserts and bolts to arrive for that. Assuming this all goes well, it’s likely that I’ll make a second one. Kind of a symmetry thing, one on each side of the french doors. I think I have more than enough Middle Atlantic TD drawers to fill two of them, and more storage is a good thing. I’d say I could always buy more Middle Atlantic TD drawers, but the price has become obscene (like 5X more than what I paid years ago). Kind sad that if someone were starting this project today, and wanted to use three 4U and three 3U drawers, they’d spend $1200 on the drawers alone. That’s more expensive than solid 5/8″ maple dovetail drawers with 100 lb. slides from quikdrawers.com and much more expensive than baltic birch drawers with slides. Of course the advantage of the Middle Atlantic TD drawers is more usable space (wood takes up a lot more space).

I stopped at Rockler on my way home and bought a can of Georgian Cherry gel stain. I don’t really imagine using this alone, since it’s redder than I’d use in the office. But I may experiment with layering it in combination with another stain, or multiple coats and then a dark glaze. The latter yields pretty results on oak in the articles I’ve read.

It looks like the relatively inexpensive hand-held 4″ wetsaw at Lowe’s gets good reviews. If I buy one, I can buy 24″ x 24″ marble-look polished porcelain tile for the top and cut it to the size I need.

Making my own office furniture: Part 2

This weekend I realized that the 3″ casters I bought for the printer stand / drawer cabinet are not ideal. Concealing them behind a skirt forces me to place them too far away from the edges, which has the potential to make the cabinet tippy. With an HP 4050TN on top, this is obviously undesirable. So I ordered the same brand of casters but in 2″ wheel diameter. I also ordered another set of the 3″ casters, and I’ll use both sets of 3″ casters on the base cabinets for the bookcases. The base cabinets are much deeper and also shorter, so tippiness shouldn’t be an issue. They also need the additional strength of the 3″ casters.

I bought and cut the pieces for the second bookcase, and also cut the slot for the back panel using the dado cutter in my table saw. I did the same for the rolling printer stand / drawer cabinet. I glued up the latter, using Loctite PL. The main reason for Loctite PL here: strength and resiliency. I also wanted a little expansion in the slots for the back panel. It’s a PITA to clean up versus yellow or white wood glue, but this cabinet’s insides won’t be visible.

I also ordered nut inserts and bolts for the casters. The nut inserts may or may not work out; they’re intended for softwood, but I’ll be using them in plywood. It’s been a long time since I’ve used this type of nut insert, so I don’t remember if they work in plywood. It all comes down to how well they handle the glue in the plywood. The plywood itself is a softwood. The idea here is that I need a doubled-up base for strength, and plywood is cheaper and more dimensionally stable than solid wood. In the last 10 years or so, when I’ve installed plate casters in wood, I’ve just used bolts and nuts with a recess cut with a Forstner bit for the bolt head and a washer. But here I want a completely concealed fastener.

I’m still debating what I want for the top of the printer stand / drawer cabinet. I know I want a marble-look porcelain tile in the center (looks nice, very durable). But I want one piece, and I haven’t found one I like that’s big enough while not also being too big for my tile saw. I’m pretty sure I can find one though. However, I haven’t yet decided how I want to trim it (species of wood), nor how thick I want it to be overall. I really love padauk, so I might go that route despite it being considerably different than oak.

Making my own office furniture

For a very long time, I’ve been using Middle Atlantic recording studio furniture in my home office. A large Edit Center Desk (ELUR) with 8U of rack space built in and a large overbridge with adjustable speaker platforms. I’ve had the desk for so long that I can’t remember what I paid for it. Today it’s about $3,800 street price (yikes!). The rack space has been used for audio gear and a pair of power distribution units with pull-out lights since day 1. I also have three MDV-R12 rack units, which are about $480 street price today. I’ve been using those for a rack mount UPS, ethernet switches, power protection and a lot of Middle Atlantic TD drawers (which are very expensive these days). I would estimate I’ve got over $8,000 worth of functional furniture for my home office. I didn’t pay that, but that’s what it would cost today.

However… I’ve never liked the look of it for my home. It is highly functional, but it’s intended for professional work space. It’s laminated MDF, black. It’s not something I want in the den of my current home. And my sweetie doesn’t want it in there either. But the den will be my office space soonish. I need new office furniture, preferably all real hardwood (and maybe metal and marble or marble-look porcelain). As much as I appreciate the recycled nature of MDF, I don’t like it for furniture. Even with a quality real wood veneer, it’s just not the same as solid hardwood.

The thing is… I still need it to be functional. I need rack space, it’s not optional. And outside of Etsy, I haven’t seen furniture with rack rails that I’d consider in the den. And no one makes exactly the cabinets and shelves I need.

Fortunately, I’m reasonably handy. And I’ve built racks before. And I have more than 30U of Middle Atlantic TD drawers (which I really like). So…

First up is some bookshelf cabinets that happen to be 19.125″ wide on the inside and hence can accept Middle Atlantic rack rails. They’re 52″ tall, 16″ deep. Drilled for 5mm shelf pins, about 15.675″ deep inside, solid red oak sides, bottom, top and shelves. The back is 1/4″ oak plywood fitted into slots in the sides, top and bottom. I want two more of these, but I’m thinking those might be shallower (around 12″). The advantage of the 15.675″ inside depth on this first bookshelf cabinet: it’s just enough to allow Middle Atlantic TD drawers on rack rails, if desired. The intent of these is that they’ll sit on a base cabinet (more on that later), starting at about desk height (30″). So the top will be at about 82″.

Here’s a picture of a test assembly.

Testing the shelves in the new bookshelf/rack cabinet.

Here’s another picture from when I was applying sanding sealer. The gap in the rear panel is to allow AC power or other cables to enter the area where I intend to put the rack rails for a Furman PL-8C, Behringer Composer PRO-XL, Behringer Ultragraph 2-channel 15-band equalizer, UMC404HD (for me) and UMC204HD for my sweetie. I’m still debating if I need more than 6U of rack space. It would not be terrible to have a 4U drawer in here to stash headphones, though given how much I’ve been working from home during the pandemic, my headphones get heavy use. Hence the current plan is to have enough space at the bottom of this cabinet to hang two pairs of headphones (with microphones).

Applying sanding sealer to the new bookshelf/rack cabinet.

Second up is similar size-wise (19.125″ wide inside and 16″ deep), but will be a rolling cabinet just for Middle Atlantic TD drawers and my main laser printer. There will be 21U of rack space. Probably three 4U and three 3U drawers, but we’ll see. It might not be a terrible idea to have a PDU in the front (with pull-out lights) and room for an ethernet switch (and my Raspberry Pi rack?) in the back. But I want a clean look here. Sort of industrial (the drawers), but in a solid hardwood cabinet. I bought 3″ casters, all swivel but two with no lock and two with total locking. A hardwood skirt will conceal the casters, floating the cabinet about 1/2″ off the floor. Why casters? Well, the older I get, the more I wish all my furniture could be easily moved (when cleaning, when needing to get to a hidden AC outlet, or data port, etc.).

The base cabinet I mentioned… I’m still in the ideas phase. Equipment-wise, I need to house UPS, PDU, ethernet switches and cooling. Today I’m doing that in a Middle Atlantic MDV-R12 to which I added rear rails for patch panels and fans. I don’t want this stuff facing the french doors to the foyer. The racks need to face north and south. I’m picturing a double-ended cabinet, and it needs to be about 66″ long (to support three of the bookshelf units). I’m imagining it being about 24″ wide, 20.5″ of that dictated by rack width. I’ll need 6 or 8 casters for this one; it will potentially be holding up a lot of weight.

This could also be two cabinets, that could be latched together or just have a bridged top. The advantage here is simply the building process; it’s much easier for me to move smaller pieces. I suppose modularity is also an advantage, but I don’t anticipate reconfiguring this stuff once it’s in place.

Desk space is also still in the idea stage. I’d really like space for the two of us to work comfortably. The hard part is not consuming the whole room with desk. Right now, I have a big ugly desktop computer. I’ve been wanting to ditch it for years, but I’m waiting on Apple for a beefier Mac Mini with Apple silicon. I can’t ditch my monitor (I need the 38″ screen), nor my keyboard, trackpad and wrist rest. My monitor is currently on an arm on the ELUR overbridge, and it’d probably be a good idea to have overbridge(s) on new desk(s); space to hide keyboard and cables (and Mac Mini?), etc. It’s also often the ideal height for speakers.

High art?

I ran across this on a list of top desks for 2021.

My first thought was “Dancing banana animated GIF meme.”. I haven’t looked up the designer nor their inspiration. But to be honest, it doesn’t matter. He/she could tell me it came from their muse. What matters… happenstance or deliberate, it’s an inanimate object that mimics an Internet meme that mimics a piece of fruit that mimics a dancing human being. Is it high art just by the number of shoulders it’s standing on? Or just a good example of “All creative work is derivative”? Can I call it anthropomorphic? Why not?

We stand on the shoulders of giants. Or on the shoulders of dancing bananas.

Not my style of desk, but bravo! Whimsical yet functional.

Time for a new desktop keyboard: the switches

I just ordered a set of Kailh Box Pink key switches from novelkeys.xyz. And I’m going to order a barebones hot swap keyboard. Why?

First, I spilled some coffee in my desktop keyboard. Maybe 1/4 cup. Which is a lot, though nowhere near what I’ve spilled on some of my buckling spring keyboards. I tore it down, doused it with 99% isopropyl alcohol, cleaned all the switches, the PCB (both sides), the case (inside and out), the keycaps. It’s working again, but it’s a reminder….

It’s my oldest CODE keyboard, Cherry MX blue switches. I like the CODE keyboard. I don’t love the price though, and I also don’t love how difficult it is to disassemble. I’ve had it apart a couple of times. And this time, one of the case tabs broke. It’s not a huge issue, but it’s annoying. If I press on the lower right corner of the keyboard case… “SQUEAK”. It’s likely more the matter that I know it’s broken than it is any sort of real annoyance, but…

I’ve never loved Cherry MX switches. My computing life began when we had some truly delightful keyboards to type on. Before the IBM Model M (very good). Even before the IBM Model F (better). Anyone truly interested in how we got where we are today would do well to do some history homework online before making your next mechanical keyboard purchase. But I can sum it up in two words: cost reductions.

I’m not going to judge that. It is what it is, and the good far outweighs the bad; cost reductions are what have allowed many of us to own desktop and laptop computers for many decades.

But… there are those of us that spend our lives at our keyboards because it’s our livelihood. And there are a LOT of us. And some of us care deeply about our human-machine interface that we spend 8+ hours at each day. And guess what? We’re not all the same. Unfortunately, we’ve mostly been saddled with only two predominant key switches for keyboards for a very long time now: various rubber dome keyboards (pretty much universally considered crummy but inexpensive), and those with Cherry MX (or something similar to Cherry MX). We do still have Unicomp making buckling spring keyboards with tooling from Lexmark (who manufactured the Model M keyboards for the North American market). And we have some new switch types (optical, hall effect, etc.). But at the moment, the keyboard world is predominantly Cherry colored.

Perhaps worse, those of us that like a switch to be both tactile and clicky have few good choices. Unicomp buckling spring is at the top for readily available and reasonably priced. But the compromises for a modern desktop are significant for a large crowd of us. Number one would be that it’s huge (including the SSK version). For some, number two would be no backlighting. And yet others want more keycap options. But it’s a long drop from the buckling spring to any MX-style switch if your goal is clicky and tactile.

I don’t hate Cherry MX blues. Nor Gateron blues. Nor many others. But… most of them feel like just what they’re designed to be. They’re not smooth (including the linears), most of them are not truly tactile, and they’re fragile (not protected from dust or liquid ingress). Most have considerable key wobble. They’re usable, I’ve just been wanting something better for a while. In a TKL (tenkeyless) keyboard with minimal footprint.

Some personal history… one of the reasons I stopped using buckling spring was just the sheer size of any of my true Model M keyboards or any of the Unicomps. The other was the activation force. I wanted something a little lighter to the touch, but still noisy. The Cherry MX blue and similar have filled the niche for me for a while. But… the scratchiness/crunchiness has always been less than ideal to me, and the sound in any board I’ve tried has been less satisfying than I’d like. I’ve not had any of the switches die on me, which is a testament to durability. But I’ve had to clean them on more than one occasion due to a small spill, to get them working again. And over time, despite the fact that they still function, their characteristics change. Some keys lose some of their clickiness. Some get louder. And out of the box, they’re not terribly consistent sound-wise. And while I’ve disassembled keyboards to clean and lube the switches… it’s very time consuming. And despite the fact that I have a pretty good hot-air rework setup, it’s very hard for me to justify spending time replacing soldered switches. I can barely justify time swapping hot-swap keys!

So… I want a more durable switch. And something smoother (less scratch/crunch) than a Cherry MX, but with a nice distinct click. And unfortunately, something that works in a PCB designed for Cherry MX since there are far and away the most options there. The Kailh White or Pink seem to fit the bill. The white are readily available, so I bought the pink just to make sure I don’t miss out on available inventory. I’ll put them in a hot-swap board with PBT keycaps and give them a test drive for a few weeks.

I know the downsides ahead of time. I had an adjustment to make when I went from buckling spring to Cherry MX blue. Buckling spring feedback and activation occur at the same time; it’s ideal. Cherry MX and related designs… most of them activate after the click. The Kailh pink and white appear to activate before the click, and they don’t have the hysteresis of the Cherry MX switches. But based on my own personal preferences which are aligned pretty closely to those who’ve reviewed the Kailh Box White and Kailh Box Pink (like Chyrosran on YouTube), I think one of these switches will make me happier than my MX blues.

Of course I could be wrong. But that’s why I’m going with an inexpensive hot-swap board for this test drive. PCB, mounting and chassis all play a significant role in how a keyboard feels and sounds. But I know many of those differences, and the goal at the moment is to pick the switches I want in my next long-term keyboard.

Facebook… go away

I don’t think I’ve ever seen such a disingenuous paragraph in a full-page newspaper ad as this one from Facebook in their ongoing attack on Apple:

Apple’s change will limit their ability to run personalized ads. To make ends meet, many will have to start charging you subscription fees or adding more in-app purchases, making the internet much more expensive and reducing high-quality free content.

Let’s be clear here. For one… there is literally NO free content on Facebook. And very little of it is high-quality. That which is, does not come from Facebook. They are not a company of journalists and writers.

Newsflash for those who’ve been under a rock for the last 25 years… the Internet has always been expensive. The real issues here:

1) who’s profiting?

2) in what currency?

3) is the transaction clear and transparent?

There are many companies profiting from the existing model of ‘free’ Internet. But it’s not small businesses. It’s Google, Facebook and a trove of others (Apple included).

On the currency and transparency… Facebook is far and away the worst here. They despise transparency. Apple wants to expose their users to what Facebook is collecting from you, the product, and let you choose whether or not you’d like to participate. You can choose to opt in. Facebook is worried that many will opt out once they understand what Facebook is doing. Not an unjust fear, but it’s yet to be seen how it will play out.

Apple’s motivation is coming from its customers. They (and I’m one of them) want these options. They’re one of the reasons we choose to buy iOS devices instead of Android devices. I don’t want targeted advertising. In fact, at this point I’ve been using the Internet for 30 years and I’m essentially blind to all online advertising; my brain has a highly-trained ad-ignoring filter. I don’t want large corporations tracking my every move online. Especially without transparency. Heaven forbid that I be willing to pay Apple for a device that allows me to protect some of my privacy!

Facebook’s motivations are at least partly coming from their customers too. But you, the end user, are NOT their customer. The advertisers are their customers. You are their product. I don’t quite get why Facebook tries to deny this; without you (the end user) and all the data they collect on you… they have no product to sell to advertisers. They’d have to change their business model. Perhaps charge a subscription fee. And for most of us… Facebook is definitely not something we’d knowingly pay ‘real’ money to use. But if you’re a Facebook user, you ARE paying for it. With your privacy and your time. And possibly your mental health. And maybe even your data plan.

And Facebook knows this to be true.

Beyond hurting apps and websites, many in the small business community say this change will be devastating for them too, at a time when they face enormous challenges. They need to be able to effectively reach the people most interested in their products and services to grow.

LMAO. “Hurting apps and websites”. Could you be more ambiguous? Oh, I see… you mean facebook.com. Sorry, I forgot for a moment that Google and Facebook have _decimated_ many small businesses as well as some large ones (news broadcasters, journalists, ad agencies, large newspapers, local sign makers…).

Again… you, the end user, are not the customer. The advertisers are the customers.

Forty-four percent of small to medium businesses started or increased their usage of personalized ads on social media during the pandemic, according to a new Deloitte study. Without personalized ads, Facebook data shows that the average small business advertiser stands to see a cut of over 60% in their sales for every dollar they spend.

In other words… once users understand what Facebook is doing, most will opt out?

Update May 12, 2021: it looks like the answer is a resounding ‘yes’. In the first week of 14.5 rollout, 96% of users are choosing to opt out. Should I feign surprise?

New IPv4 and IPv6 container templates

I’ve spent a little bit of time working on some new slimmed-down C++ containers keyed by IPv4 addresses, IPv6 address, IPv4 prefixes and IPv6 prefixes. The containers that are keyed by prefixes allow longest-match searching by address, as would be expected.

My main objective here was to minimize the amount of code I need to maintain, by leveraging the C++ standard library and existing classes and class templates in libDwm. A secondary objective was to make sure the containers are fast enough for my needs. A third objective was to make the interfaces thread safe.

I think I did OK on the minimal code front. For example, DwmIpv4PrefixMap.hh is only 102 lines of code (I haven’t added I/O functionality yet). DwmIpv6PrefixMap.hh is 185 lines of code, including I/O functionality. Obviously they leverage existing code (Ipv4Prefix, Ipv6Prefix, et. al.).

The interfaces are thread safe. I’m in the process of switching them from mutex and lock_guard to shared_mutex and shared_lock/unique_lock.

Performance-wise, it looks pretty good. I’m using prefix dumps from routeviews to have realistic data for my unit tests. On my Threadripper 3960X development machine running Ubuntu 20.04:


 % ./TestIpv4AddrMap -p
 831,915 addresses, 7,380,956 inserts/sec
 831,915 addresses, 16,641,961 lookups/sec
 831,915 addresses, 9,032,736 removals/sec
 831,915 addresses, 8,249,196 inserts/sec (bulk lock)
 831,915 addresses, 54,097,737 lookups/sec (bulk lock)
 831,915 addresses, 9,489,272 removals/sec (bulk lock)
 831,918/831,918 passed

 % ./TestIpv4PrefixMap -p
 901,114 prefixes, 6,080,842 prefix inserts/sec
 901,114 prefixes, 14,639,881 prefix lookups/sec
 901,114 addresses, 5,105,259 longest match lookups/sec
 901,114 prefixes, 6,378,710 prefix inserts/sec (bulk lock)
 901,114 prefixes, 25,958,230 prefix lookups/sec (bulk lock)
 901,114 addresses, 5,368,727 longest match lookups/sec (bulk lock)
 1,802,236/1,802,236 passed

 % ./TestIpv6AddrMap -p
 104,970 addresses, 11,360,389 inserts/sec
 104,970 addresses, 15,206,431 lookups/sec
 104,970 addresses, 9,159,685 removals/sec
 104,970 addresses, 12,854,518 inserts/sec (bulk lock)
 104,970 addresses, 20,434,105 lookups/sec (bulk lock)
 104,970 addresses, 10,302,286 removals/sec (bulk lock)
 104,976/104,976 passed

 % ./TestIpv6PrefixMap -p
 110,040 prefixes, 11,181,790 prefix lookups/sec
 110,040 prefixes, 1,422,403 longest match lookups/sec
 440,168/440,168 passed


What is ‘bulk lock’? The interfaces allow one to get a shared or unique lock and then perform multiple operations while holding the lock. As seen above, this doesn’t make a huge difference for insertion or removal of entries, where the time is dominated by operations other than locking and unlocking. It does make a significant difference for exact-match searches. One must be careful using the bulk interfaces to avoid deadlock, of course. But they are useful in some scenarios.

The best part, IMHO, is that these are fairly thin wrappers around std::unordered_map. Meaning I don’t have my own hash table or trie code to maintain and I can count on std::unordered_map behaving in a well-defined manner due to it being part of the C++ standard library. It is not the fastest means of providing longest-match lookups. However, from my perspective as maintainer… it’s a small bit of code, and fast enough for my needs.

Threadripper 3960X: the birth of ‘thrip’

I recently assembled a new workstation for home. My primary need was a machine for software development, including deep learning. This machine is named “thrip”.

Having looked hard at my options, I decided on AMD Threadripper 3960X as my CPU. A primary driver was of course bang for the buck. I wanted PCIe 4.0, at least 18 cores, at least 4-channel RAM, the ability to utilize 256G or more of RAM, and to stay in budget.

By CPU core count alone, the 3960X is over what I needed. On the flip side, it’s constrained to 256G of RAM, and it’s also more difficult to keep cool than most CPUs (280W TDP). But on price-per-core, and overall performance per dollar, it was the clear winner for my needs.

Motherboard-wise, I wanted 10G ethernet, some USB-C, a reasonable number of USB-A ports, room for 2 large GPUs, robust VRM, and space for at least three NVMe M.2 drives. Thunderbolt 3 would have been nice, but none of the handful of TRX40 boards seem to officially support it (I don’t know if this is an Intel licensing issue or something else). The Gigabyte board has the header and Wendell@Level1Techs seems to have gotten it working, but I didn’t like other aspects of the Gigabyte TRX40 AORUS EXTREME board (the XL-ATX form factor, for example, is still limiting in terms of case options).

I prefer to build my own workstations. It’s not due to being particularly good at it, or winding up with something better than I could get pre-built. It’s that I enjoy the creative process of selecting parts and putting it all together.

I had not assembled a workstation in quite some time. My old i7-2700K machine has met my needs for most of the last 8 years. And due to a global pandemic, it wasn’t a great time to build a new computer. The supply chain has been troublesome for over 6 months now, especially for some specific parts (1000W and above 80+ titanium PSUs, for example). We’ve also had a huge availability problem for the current GPUs from NVIDIA (RTX 3000 series) and AMD (Radeon 6000 series). And I wasn’t thrilled about doing a custom water-cooling loop again, but I couldn’t find a worthy quiet cooling solution for Threadripper and 2080ti without going custom loop. Given the constraints, I wound up with these parts as the guts:

  • Asus TRX40 ROG Zenith II Extreme Alpha motherboard
  • AMD Threadripper 3960X CPU (24 cores)
  • 256 gigabytes G.Skill Trident Z Neo Series RGB DDR4-3200 CL16 RAM (8 x 32G)
  • EVGA RTX 2080 Ti FTW3 Ultra GPU with EK Quantum Vector FTW3 waterblock
  • Sabrent 1TB Rocket NVMe 4.0 Gen4 PCIe M.2 Internal SSD
  • Seasonic PRIME TX-850, 850W 80+ Titanium power supply
  • Watercool HEATKILLER IV PRO for Threadripper, pure copper CPU waterblock

It’s all in a Lian Li PC-O11D XL case. I have three 360mm radiators, ten Noctua 120mm PWM fans, an EK Quantum Kinetic TBE 200 D5 PWM pump, PETG tubing and a whole bunch of Bitspower fittings.

My impressions thus far: it’s fantastic for Linux software development. It’s so nice to be able to run ‘make -j40‘ on large C++ projects and have them complete in a timely manner. And thus far, it runs cool and very quiet.

An ode to NSFNET and ANSnet: a simple NMS for home

A bit of history…

I started my computing career at NSFNET at the end of 1991. Which then became ANSnet. In those days, we had a home-brewed network monitoring system. I believe most/all of it was originally the brainchild of Bill Norton. Later there were several contributors; Linda Liebengood, myself, others. The important thing for today’s thoughts: it was named “rover”, and its user interface philosophy was simple but important: “Only show me actionable problems, and do it as quickly as possible.”

To understand this philosophy, you have to know something about the primary users: the network operators in the Network Operations Center (NOC). One of their many jobs was to observe problems, perform initial triage, and document their observations in a trouble ticket. From there they might fix the problem, escalate to network engineering, etc. But it wasn’t expected that we’d have some omniscient tool that could give them all of the data they (or anyone else) needed to resolve the problem. We expected everyone to use their brains, and we wanted our primary problem reporter to be fast and as clutter-free as possible.

For decades now, I’m spent a considerable amount of time working at home. Sometimes because I was officially telecommuting, at other times just because I love my work and burn midnight hours doing it. As a result, my home setup has become more complex over time. I have 10 gigabit ethernet throughout the house (some fiber, some Cat6A).  I have multiple 10 gigabit ethernet switches, all managed.  I have three rackmount computers in the basement that run 7×24.  I have ZFS pools on two of them, used for nightly backups of all networked machines, source code repository redundancy, Time Machine for my macOS machines, etc.  I run my own DHCP service, an internal DNS server, web servers, an internal mail server, my own automated security software to keep my pf tables current, Unifi, etc.  I have a handful of Raspberry Pis doing various things.  Then there’s all the other devices: desktop computers in my office, a networked laser printer, Roku, AppleTV, Android TV, Nest thermostat, Nest Protects, WiFi access points, laptops, tablet, phone, watch, Ooma, etc.  And the list grows over time.

Essentially, my home has become somewhat complex.  Without automation, I spend too much time checking the state of things or just being anxious about not having time to check everything at a reasonable frequency.  Are my ZFS pools all healthy?  Are all of my storage devices healthy?  Am I running out of storage space anywhere?  Is my DNS service working?  Is my DHCP server working?  My web server?  NFS working where I need it?  Is my Raspberry Pi garage door opener working?  Are my domains resolvable from the outside world?  Are the cloud services I use working?  Is my Internet connection down?  Is there a guest on my network?  A bandit on my network?  Is my printer alive?  Is my internal mail service working?  Are any of my UPS units running on battery?  Are there network services running that should not be?  What about the ones that should be, like sshd?

I needed a monitoring system that worked like rover; only show me actionable issues.  So I wrote my own, and named it “mcrover”.  It’s more of a host and service monitoring system than a network monitoring system, but it’s distributed and secure (using ed25519 stuff in libDwmAuth).  It’s modern C++, relatively easy to extend, and has some fun bits (ASCII art in the curses client when there are no alerts, for example).  Like the old Network Operations Center, I have a dedicated display in my office that only displays the mcrover Qt client, 24 hours a day.  Since most of the time there are no alerts to display, the Qt client toggles between a display of the next week’s forecast and a weather radar image when there are no alerts.  If there are alerts, the alert display will be shown instead, and will not go away until there are no alerts (or I click on the page switch in the UI).  The dedicated display is driven by a Raspberry Pi 4B running the Qt client from boot, using EGLFS (no X11).  The Raspberry Pi4 is powered via PoE.  It is also running the mcrover service, to monitor local services on the Pi as well as many network services.  In fact the mcrover service is running on every 7×24 general purpose computing device.  mcrover instances can exchange alerts, hence I only need to look at one instance to see what’s being reported by all instances.

This has alleviated me of a lot of sys admin and network admin drudgery.  It wasn’t trivial to implement, mostly due to the variety (not the quantity) of things it’s monitoring.  But it has proven itself very worthwhile.  I’ve been running it for many months now, and I no longer get anxious about not always keeping up with things like daily/weekly/monthly mail from cron and manually checking things.  All critical (and some non-critical) things are now being checked every 60 seconds, and I only have my attention stolen when there is an actionable issue found by mcrover.

So… an ode to the philosophy of an old system.  Don’t make me plow through a bunch of data to find the things I need to address.  I’ll do that when there’s a problem, not when there isn’t a problem.  For 7×24 general purpose computing devices running Linux, macOS or FreeBSD, I install and run the mcrover service and connect it to the mesh.  And it requires very little oomph; it runs just fine on a Raspberry Pi 3 or 4.

So why the weather display?  It’s just useful to me, particularly in the mowing season where I need to plan ahead for yard work.  And I’ve just grown tired of the weather websites.  Most are loaded with ads and clutter.  All of them are tracking us.  Why not just pull the data from tax-funded sources in JSON form and do it myself?  I’ve got a dedicated display which doesn’t have any alerts to display most of the time, so it made sense to put it there.

The Qt client using X11, showing the weather forecast.

mcrover Qt client using X11, showing the weather forecast

The Qt client using EGLFS, showing the weather radar.

The curses client, showing ASCII art since there are no alerts to be shown.

mcrover curses client with no alerts.