Category: FreeBSD

My custom garage door opener appliance is running FreeBSD 11.0-ALPHA5 on a Raspberry Pi 2B. It has worked fine for about 8 years now. However, I want to migrate it to FreeBSD 13.2-STABLE and from libDwmAuth to libDwmCredence. And just bring the code current.

The tricky part is that I never quite finished packaging up my device driver for the rotary encoders, and it was somewhat experimental (hence the alpha release of FreeBSD). But as of today it appears I have the rotary encoder device drivers working fine on FreeBSD 13.2-STABLE on a Raspberry Pi 4B. The unit tests for libDwmPi are passing, and I’m adding to them and doing a little cleanup so I’ll be able to maintain it longer-term.

I should note that the reason I went with FreeBSD at the time was pretty simple: the kernel infrastructure for what I needed to do was significantly better versus linux. That may or may not be true today, but for the moment I have no need to look at doing this on linux. The only non-portable code here is my device driver, and it’s relatively tiny (including boilerplate stuff).

Looking back at this project, I should have made a few more hardware-wise. The Raspberry Pi 2B is more than powerful enough for the job, and given that I put it inside a sealed enclosure, the lower power consumption versus a 4B is nice. I’m pretty sure my mom would appreciate one of these, if just by virtue of being able to open her garage doors with her phone or watch. The hardware (the Pi and the HAT I created) has been flawless, and I’ve had literally zero issues despite it being in a garage with no climate control (so it’s seen plenty of -10F and 95F days). It just works.

However, today I could likely do this in a smaller enclosure, thanks to PoE HATs. Unfortunately not the official latest Raspberry Pi PoE HAT because its efficiency is abysmal (generates too much heat). If I bump the Pi to a 4B, I’ll probably stick with a separate PoE splitter (fanless). I’ll need a new one since the power connector has changed.

The arguments for moving to a Pi 4B:

• future-proofing. If I want to build another one, I’m steered toward the Pi 4B simply because it’s what I can buy and what’s current.
• faster networking (1G versus 100M)
• more oomph for compiling C and C++ code locally
• Some day, the Pi 2B is going to stop working. I’ve no idea when that day might be, but 8 years in Michigan weather seems like it has probably taken a significant toll. On the other hand it could last another 20 years. There are no electrolytic capacitors, I’m using it headless, and none of the USB ports are in use.

The arguments against it:

• higher power consumption, hence more heat
• the Pi 2B isn’t dead yet

I think it’s pretty clear that during this process, I should try a Pi 4B. The day will come when I’ll have to abandon the 2B, and I’d rather do it on my timeline. No harm in keeping the 2B in a box while I try a 4B. Other than the PoE splitter, it should be a simple swap. Toward that end, I ordered a 4B with 4G of RAM (I don’t need 8G of RAM here). I still need to order a PoE splitter, but I can probably scavenge an original V2 PoE HAT from one of my other Pis and stack with stacking headers.

Over the weekend I started building FreeBSD 13.2-STABLE (buildworld) on the Pi 2B and as usual hit the limits. The problem is that 1G of RAM isn’t sufficient to utilize the 4 cores. It’s terribly slow even when you can use all 4 cores, but if you start swapping to a microSD card… it takes days for 'make buildworld‘ to finish. And since I have a device driver I’m maintaining for this device, it’s expected that I’ll need to rebuild the kernel somewhat regularly and also build the world occasionally. This is the main motivation for bumping to a Raspberry Pi 4B with 4G of RAM. It is possible it’ll still occasionally start swapping with a ‘make -j4 buildworld‘ , but the cores are faster and I don’t frequently see a single instance of the compiler or llvm-tblgen go over 500M, but it does happen. I think 4G is sufficient to avoid swapping during a full build.

Update Aug 26, 2023: duh, a while after I first created mcpigdo, it became possible to do what I need to do with the rotary encoders from user space. With FreeBSD 13.2, I can configure interrupts on the GPIO pins and be notified via a number of means. I’m going to work on changing my code to not need my device driver. This is good news since I’ve had some problems with my very old device driver despite refactoring, and I don’t have time to keep maintaining it. Moving my code to user space will make it more portable going forward, though it’ll still be FreeBSD-only. It will also allow for more flexibility.

I installed a new Eaton 5PX1500RT in my basement rack this week. I’d call it “planned, sort of…”. My last Powerware 5115 1U UPS went into an odd state which precipitated the new purchase. However, it was on my todo list to make this change.

I already own an Eaton 5PX1500RT, which I bought in 2019. I’ve been very happy with it. It’s in the basement rack, servicing a server, my gateway, ethernet switches and broadband modem. As is my desire, it is under 35% load.

The Powerware 5115 was servicing my storage server, and also under 35% load. This server has dual redundant 900W power supplies.

Installation of the new UPS… no big deal. install the ears, install the rack rails, rack the UPS.

Shut down the devices plugged into the old UPS, plug them in to the new UPS. Boot up, check each device.

Install the USB cable from the UPS to the computer that will monitor the state of the UPS. Install Network UPS Tools (nut) on that computer. Configure it, start it, check it.

This week, at this step things got… interesting.

I was monitoring the old Powerware 5115 from ‘ria’. ‘ria’ is a 1U SuperMicro server with a single Xeon E3-1270 V2. It has four 1G ethernet ports and a Mellanox 10G SFP+ card. Two USB ports. And a serial port which has been connected to the Powerware 5115 for… I don’t know, 8 years?

I can monitor the Eaton 5PX1500RT via a serial connection. However, USB is more modern, right? And the cables are less unwieldy (more wieldy). So I used the USB cable.

Trouble started here. The usbhid-ups driver did not reliably connect to the UPS. When it did, it took a long time (in excess of 5 seconds, an eternity in computing time). ‘ria’ is running FreeBSD 12.3-STABLE on bare metal.

I initially decided that I’d deal with it this weekend. Either go back to using a serial connection or try using a host other than ‘ria’. However…

I soon noticed long periods where mcrover was displaying alerts for many services on many hosts. Including alerts for local services, whose test traffic does not traverse the machine I touched (‘ria’). And big delays when using my web browser. Hmm…

Poking around, I seemed to only be able to reliably reproduce a network problem by pinging certain hosts with ICMPv4 from ria and observing periods where the round trip time would go from .05 milliseconds to 15 or 20 seconds. No packets lost, just periods with huge delays. These were all hosts on the same 10G ethernet network. ICMPv6 to the same hosts: no issues. Hmm…

I was eventually able to correlate (in my head) what I was seeing in the many mcrover alerts. On the surface, many didn’t involve ‘ria’. But under the hood they DO involve ‘ria’ simply because ‘ria’ is my primary name server. So, for example, tests that probe via both IPv6 and IPv4 might get the AAAA record but not the A record for the destination, or vice versa, or neither, or both. ‘ria’ is also the default route for these hosts. I honed in on the 10G ethernet interface on ‘ria’.

What did IPV4 versus IPv6 have to do with the problem? I don’t know without digging through kernel source. What was happening: essentially a network ‘pause’. Packets destined for ‘ria’ were not dropped, but queued for later delivery. As many as 20 seconds later! The solution? Unplug the USB cable for the UPS and kill usbhid-ups. In the FreeBSD kernel, is USB hoarding a lock shared with part of the network stack?

usbhid-ups works from another Supermicro server running the same version of FreeBSD. Different hardware (dual Xeon L5640). Same model of UPS with the same firmware.

This leads me to believe this isn’t really a lock issue. It’s more likely an interrupt routing issue. And I do remember that I had to add hw.acpi.sci.polarity="low" to /boot/loader.conf on ‘ria’ a while ago to avoid acpi0 interrupt storms (commented out recently with no observed consequence). What I don’t remember: what were all the issues I found that prompted me to add that line way back when?

Anyway… today’s lesson. Assume the last thing you changed has high probability of cause, even if there seems to be no sensible correlation. My experience this week: “Unplug the USB connection to the UPS and the 10G ethernet starts working again. Wait, what?!”.

A bit of history…

I started my computing career at NSFNET at the end of 1991. Which then became ANSnet. In those days, we had a home-brewed network monitoring system. I believe most/all of it was originally the brainchild of Bill Norton. Later there were several contributors; Linda Liebengood, myself, others. The important thing for today’s thoughts: it was named “rover”, and its user interface philosophy was simple but important: “Only show me actionable problems, and do it as quickly as possible.”

To understand this philosophy, you have to know something about the primary users: the network operators in the Network Operations Center (NOC). One of their many jobs was to observe problems, perform initial triage, and document their observations in a trouble ticket. From there they might fix the problem, escalate to network engineering, etc. But it wasn’t expected that we’d have some omniscient tool that could give them all of the data they (or anyone else) needed to resolve the problem. We expected everyone to use their brains, and we wanted our primary problem reporter to be fast and as clutter-free as possible.

For decades now, I’m spent a considerable amount of time working at home. Sometimes because I was officially telecommuting, at other times just because I love my work and burn midnight hours doing it. As a result, my home setup has become more complex over time. I have 10 gigabit ethernet throughout the house (some fiber, some Cat6A).  I have multiple 10 gigabit ethernet switches, all managed.  I have three rackmount computers in the basement that run 7×24.  I have ZFS pools on two of them, used for nightly backups of all networked machines, source code repository redundancy, Time Machine for my macOS machines, etc.  I run my own DHCP service, an internal DNS server, web servers, an internal mail server, my own automated security software to keep my pf tables current, Unifi, etc.  I have a handful of Raspberry Pis doing various things.  Then there’s all the other devices: desktop computers in my office, a networked laser printer, Roku, AppleTV, Android TV, Nest thermostat, Nest Protects, WiFi access points, laptops, tablet, phone, watch, Ooma, etc.  And the list grows over time.

Essentially, my home has become somewhat complex.  Without automation, I spend too much time checking the state of things or just being anxious about not having time to check everything at a reasonable frequency.  Are my ZFS pools all healthy?  Are all of my storage devices healthy?  Am I running out of storage space anywhere?  Is my DNS service working?  Is my DHCP server working?  My web server?  NFS working where I need it?  Is my Raspberry Pi garage door opener working?  Are my domains resolvable from the outside world?  Are the cloud services I use working?  Is my Internet connection down?  Is there a guest on my network?  A bandit on my network?  Is my printer alive?  Is my internal mail service working?  Are any of my UPS units running on battery?  Are there network services running that should not be?  What about the ones that should be, like sshd?

I needed a monitoring system that worked like rover; only show me actionable issues.  So I wrote my own, and named it “mcrover”.  It’s more of a host and service monitoring system than a network monitoring system, but it’s distributed and secure (using ed25519 stuff in libDwmAuth).  It’s modern C++, relatively easy to extend, and has some fun bits (ASCII art in the curses client when there are no alerts, for example).  Like the old Network Operations Center, I have a dedicated display in my office that only displays the mcrover Qt client, 24 hours a day.  Since most of the time there are no alerts to display, the Qt client toggles between a display of the next week’s forecast and a weather radar image when there are no alerts.  If there are alerts, the alert display will be shown instead, and will not go away until there are no alerts (or I click on the page switch in the UI).  The dedicated display is driven by a Raspberry Pi 4B running the Qt client from boot, using EGLFS (no X11).  The Raspberry Pi4 is powered via PoE.  It is also running the mcrover service, to monitor local services on the Pi as well as many network services.  In fact the mcrover service is running on every 7×24 general purpose computing device.  mcrover instances can exchange alerts, hence I only need to look at one instance to see what’s being reported by all instances.

This has alleviated me of a lot of sys admin and network admin drudgery.  It wasn’t trivial to implement, mostly due to the variety (not the quantity) of things it’s monitoring.  But it has proven itself very worthwhile.  I’ve been running it for many months now, and I no longer get anxious about not always keeping up with things like daily/weekly/monthly mail from cron and manually checking things.  All critical (and some non-critical) things are now being checked every 60 seconds, and I only have my attention stolen when there is an actionable issue found by mcrover.

So… an ode to the philosophy of an old system.  Don’t make me plow through a bunch of data to find the things I need to address.  I’ll do that when there’s a problem, not when there isn’t a problem.  For 7×24 general purpose computing devices running Linux, macOS or FreeBSD, I install and run the mcrover service and connect it to the mesh.  And it requires very little oomph; it runs just fine on a Raspberry Pi 3 or 4.

So why the weather display?  It’s just useful to me, particularly in the mowing season where I need to plan ahead for yard work.  And I’ve just grown tired of the weather websites.  Most are loaded with ads and clutter.  All of them are tracking us.  Why not just pull the data from tax-funded sources in JSON form and do it myself?  I’ve got a dedicated display which doesn’t have any alerts to display most of the time, so it made sense to put it there.

The Qt client using X11, showing the weather forecast.

The Qt client using EGLFS, showing the weather radar.

The curses client, showing ASCII art since there are no alerts to be shown.