Posted on

Mac Studio M1 Ultra: The Decision

I’ve needed a macOS desktop for many years. My hackintosh, built when Apple had no current hardware to do what I needed to do, is more than 10 years of age. It’s my primary desktop. It’s behind on OS updates (WAY behind). It’s old. To be honest, I’m quite surprised it still runs at all. Especially the AIO CPU cooler.

The urgency was amplified when Apple silicon for macOS hit the streets. Apple is in transition, and at some point in the future, there will be no support for macOS on Intel. They’ve replaced Intel in the laptops, there’s an M1 iMac and an M1 mini, and now the Mac Studio. We’ve yet to see an Apple silicon Mac Pro, and while I’m sure it’s coming, I can’t say when nor anything about the pricing. If I assume roughly 2X multicore performance versus the M1 Ultra SOC, plus reasonable PCIe expansion, it’ll likely be out of my price range.

Fortunately, for today and the foreseeable future, the Mac Studio fits my needs. In terms of time == money, my main use is compiling C++ and C code. While single-core performance helps, so does multi-core for any build that has many units that can be compiled in parallel. So, for example, the Mac Studio with M1 Ultra has 20 CPU cores. Meaning my builds can compile 20 compilation units in parallel. Obviously there are points in my builds where I need to invoke the archiver or the linker on the result of many compiles. Meaning that for parts of the build, we’ll be single-core for a short period unless the tool itself (archiver, linker, etc.) uses multiple threads.

It’s important to note that a modern C++ compiler is, in general, a memory hog. It’s pretty common for me to see clang using 1G of RAM (resident!). Run 20 instances, that’s 20G of RAM. In other words, the 20 cores need at least 1G each to run without swapping. Add on all the apps I normally have running, and 32G is not enough RAM for me to make really effective use of the 20 cores, day in and day out.

So 64G would my target. And given that the CPU and GPU share that memory, that’s a good target for me. However…

Availability of Mac Studios with the exact configuration I wanted has been abysmal since… well… introduction. I wanted M1 Ultra with 64-core GPU, 64G RAM, 2TB storage. Apple’s lead time for this or anything close: 12 weeks. I’m assuming that a lot of this is the ongoing supply chain issues, COVID and possibly yield issues for the M1 Ultra. Apple is missing out on revenue here, so it’s not some sort of intentional move on their part, as near as I can tell. While I think there are M2 Pro and M2 Max on the horizon for the MacBook Pro (I dunno, 1H2023?), I think it’ll be a year before I see something clearly better for my use than the M1 Ultra. I can’t wait a year, unfortunately. I also can’t wait 3 months.

In fact, since I’m closing in on finishing the den, and need to move my office there, this is now urgent just from a space and aesthetics perspective. I intentionally designed the desk overbridges in the den to comfortably accommodate a Mac Studio (or Mac Mini) underneath either side. I DON’T want my hackintosh in this room! I want quiet, aesthetically pleasing, small, inconspicuous, efficient, and not a major source of heat. I need 10G ethernet. Fortunately, the Mac Studio ticks all of the boxes.

Today I picked up what was available, not exactly what I wanted. It’s an open box and hence $500 off: a Mac Studio with M1 Ultra, 64-core GPU, 128G of RAM and 1TB storage. The only thing from my wishlist not met here: 2TB storage. However, I’m only using 45% of the space on my 1TB drive in my hackintosh, and I haven’t tried to clean up much. I don’t keep music and movies on my desktop machine, but if I wanted to with the Mac Studio, I could plug in Thunderbolt 4 storage.

I’m much more excited about moving into the den than I am about the new computer. That’s unlikely to change, since the den remodeling is the culmination of a lot of work. And I know that I’m going to have to fiddle to make the new Mac Studio work well with my Dell U3818DW display. Assuming that goes well, I’m sure I’ll have a positive reaction to the Mac Studio. The Geekbench single-core scores are double that of my hackintosh. The multi-core scores are 7 times higher. This just gives me confidence that I’ll notice the speed when using it for my work. Especially since the storage is roughly a decimal order of magnitude faster. The 2TB is faster, but the jump will be huge from SATA to NVMe for my desktop. I notice this in my Threadripper machine and I’ll notice it here.

My main concern long-term is the cooling system. Being a custom solution from Apple, I don’t have options when the blower fans fail. Hopefully Apple will extend repairability beyond my first 3 years of AppleCare+. I like keeping my main desktop for more than 3 years. While in some ways it’s the easiest one to replace since it’s not rackmounted and isn’t critical to other infrastructure, it’s also my primary interface to all of my other machines: the Threadripper workstation for Linux and Windows development, my network storage machine, my web server, my gateway, and of course the web, Messages, email, Teams, Discord, etc. It saves me time and money if it lasts awhile.

Posted on

Grout color: sometimes you get lucky

I hemmed and hawed for a long time over the grout color for the den. The tiles are various colors (on purpose), with a reclaimed wood look. There are off-white, tan, brown, gray and charcoal tiles with different patterns in each color.

My typical go-to grout color is natural gray. Mainly because it works with many things, doesn’t grab attention, and is forever available from various companies (Laticrete, Mapei, TEC, Custom Building Products, etc.). Meaning you’ll be able to find it if you ever need a patch repair.

The den has wood paneled walls. Though it’s just plywood and moulding, the plywood is designed to look like edge-glued boards. It’s a stained red oak in color (probably originally ‘honey oak’), but the grain tells me it’s not oak. However, I wanted something that went _reasonably_ well with the walls. In other words, a brown grout of some type.

I chose SpectraLOCK Pro Premium from Laticrete because I’ve used it before and it’s a very good grout. It’s harder to install than a cementitious or ready-to-spread urethane or acrylic grout, but it’s worth the effort. It’s essentially stain proof after curing, it’s very strong, and in my experience it is very color-consistent from box to box (especially if you buy ‘Part C’ all at once to get boxes from the same batch).

The problem is that no one local stocks it. None of the big box stores carry it at all, except Floor & Decor which isn’t close to me. But the worst part is that you just can’t trust color swatches. While my main computer monitor is very color accurate, if I look at color swatches on various web sites for the color I chose, they vary dramatically. And the swatch on the grout box itself doesn’t quite match the cured grout.

I chose ‘#55 Tawny’, after thinking for a long time that I’d use Chocolate Truffle. I’m so glad I waited and gambled on the Tawny; it was the right choice. Mostly by sheer luck.

Posted on

Using SpectraLOCK Pro Premium grout

It’s been a while since I’ve used Laticrete SpectraLOCK Pro Premium grout on textured tiles. I’m in the process of grouting the den floor, and the tiles are textured. The secrets to success as a DIYer with this grout…

  • Use the mini kits, not the full kits. The grout firms up fairly quickly, and becomes hard to work with in as little as 15 minutes (depending on temperature and humidity and the difficulty of the joints). You don’t want to mix more than you can use in about 15 minutes.
  • Use a 2-gallon bucket for the cleaning packets as prescribed, but use a separate 5-gallon bucket of clean water to rinse out the sponge before dunking it and wringing it in the cleaning solution again. This will put most of the epoxy into the 5-gallon bucket instead of your cleaning solution, and greatly reduce gummy build-up in the sponge.
  • Pay attention to timing. If you start the initial washdown too soon, you’ll just spread the epoxy around and probably nick up the joints. But don’t wait too long. I wait until it doesn’t stick to my finger before starting the initial wash down.
  • A white scrub pad is very useful on the second washdown. It will allow you to work the joints as needed, and loosen up epoxy on the tiles. I then use a sponge with a microfiber backing. One swipe with the sponge side, flip it over, one swipe with the microfiber.
  • As a final step I wipe down with a cheap microfiber cloth.

I’m using a 30% acetic acid (vinegar) jug to clean my sponges afterward, mixed in an appropriate ratio with water. Then rinse thoroughly. Since the mini kits come with their own sponge, I use that sponge for the initial washdown and then throw it away. It feels a little bit wasteful, but to be honest, you’re not going to get good use out of it beyond one mini kit.

Posted on

Making my own office furniture: part 7

I bought 11/32″ plywood at Home Depot and another quart of Minwax satin wipe-on polyurethane at Menard’s.

The plywood will be used as underlayment for the porcelain tiles inset in the top of the under-desk rack cabinet. I cut it to size, checked the fitment, and marked it for screws. It will be screwed and glued (TiteBond III) to the existing plywood of the top.

The tiles in the top are 17+7/8″ square. I’m using one whole one and one that I cut to 8+1/4″ width tonight on the tile saw. I slightly beveled the edge after cutting. The plan is to use Loctite PL Premium MAX to adhere the tiles to the underlayment. This isn’t a floor, and in fact in its intended use it will never have anything on top of it. But if we ever decide we don’t want it to live under the desks, it will make a nice robust top. The tiles are PEI 4, which should far outlive me for a table top. I like porcelain table tops, since they’re impervious to water and heat. If they’re PEI 4 or PEI 5 porcelain, they’re also nearly impossible to scratch. Easy to clean, sanitary, etc. I’m not going to use epoxy grout here, but that’s an option when desired.

I think this cabinet is quite nice for it’s intended purpose. I went a little overboard in the mix of materials and the front door design, but that’s the advantage of designing and building your own stuff from raw materials and parts. You can take your time and get exactly what you want, without compromising in areas that matter to you.

Looking back at this, there’s an unconventional mix of materials and components in this cabinet design. It’s a computer rack designed for a specific set of gear arranged in a particular manner, so it has rack rails front and rear. It has casters. Neither of those are unconventional. But then the cabinet itself is made of edge-glued solid oak panels (not steel, not MDF). It has a skirt to hide the casters. It has a very unconventional front door, to show the gear in a subdued manner (tinted scratch-resistant polycarbonate) and vent from the sides, top and bottom of the door. The door has solid brass butler tray hinges. It has marine grade solid brass hold down latches to hold it shut. It has a guide pin and leaded bronze bushing to make it self-center every time it’s closed. The ventilation holes are covered with filters inspired by an antique pie safe I saw in a bakery 20 years ago. The bottom of the inside is covered with a piece of UHMW so it’ll be easy to get my UPS in and out without using rack slides. The top is porcelain framed in solid oak.

You can’t go out and buy this kind of thing at a store. You could commission it, but given the amount of time I spent on the design in SketchUp, it’d be expensive. But you can build it. It requires patience, persistence, and multiple skills (and tools). But that last part… we learn by doing. If you’re not afraid to fail, you can build just about anything. And as a software engineer, I really appreciate the fact that I can spend no money on materials until I have the design nearly completed in a 3D drawing on my computer. The execution of the design just follows the drawing.

The desks I’m building follow a similar pattern. The Delrin (acetal) feet aren’t typical. Nor is the porcelain top (I couldn’t find anything on Google with respect to building a desk with an inset porcelain slab). Nor is the combination of joinery I used (dowels, pocket holes, guide pins and leaded bronze bushings, dowel nuts with long bolts, threaded inserts).

Porcelain tiles of the size I’m using are relatively new here in the U.S., and given that I’m not running stringers under the top, could be considered ‘risky’. But my fear has been allayed by handling the tiles. Despite the fact that they’re 60″ long and 30″ wide but only 6.5mm thick, I’ve not broken one just carrying them around like pieces of plywood. And they will flex a bit without cracking. I do have suction cups for placing them, but I haven’t needed them for general handling. And it’s desk tops, not workbenches. I won’t be hammering on them, nor putting a ton of weight on them. There’s 1.25″ of plywood underneath, and 1″ thick solid oak bordering. The top will be on a very strong base. I don’t think the porcelain is going to crack on me. And it’s sanitary, PEI 5 (nearly scratchproof), impervious to any fluids I’d have on my desk (coffee, water, juice), and easy to clean. It looks like polished marble but it’s manmade (didn’t require carving up the planet). It can’t be dented by writing instruments. It won’t be bothered when I spill candle wax on it. My watchbands won’t scratch it. It doesn’t care about sweat, and in fact it’s a comfortably cool surface to rest your forearms on. I can adhere things to it (cable guides, phone dock, etc.) and later remove them with no damage whatsoever.

Risky? Perhaps. But from risks come rewards and learning experiences.

Posted on

Making my own office furniture: part 6

I continued making filter panels for the under-desk rack. The frames are 3/4″ wide solid oak, with a recess cut in the back to hold the screen. The screen is 22 squares per linear inch (484 squares per square inch) 316 stainless steel. It’s supported by 1/4″ square galvanized mesh, just to make it more difficult to damage the stainless steel screen when vacuuming them clean. The screen and mesh are held in the recess with a bead of hot glue which won’t be visible.

Below is a picture of one of the side panels. Obviously I’m not looking to keep fine dust from passing. I don’t want a lot of restriction. I just want to keep things like pet hair and the like out of the fans in the gear in the rack. The stainless steel screen is 70% open.

I’ve finished assembling three of the four filters. I need more stainless steel screen to complete the final filter. It’s McMaster-Carr 9230T51.

Posted on

Making my own office furniture: part 5

One of the desk bases is complete. The second one is built but needs stain and topcoat.

I’ve been working on the top for one of the desks.  As a reminder, here’s the desk design from SketchUp.

The base of the top is 3/4″ oak plywood. Above that is a piece of 1/2″ BCX plywood underneath the porcelain insert. The border of the top is 1″ thick solid oak, with four 5/16″ diameter oak dowels at each joint. The border sits atop the oak plywood. Hence the total top thickness is roughly 1.75″. Solid oak trim pieces cover the edges of the oak plywood.

Below is a picture from when I was testing the fitment of the porcelain insert.  There’s a 3/16″ gap between the porcelain and the frame.  This is to allow for flexible grout or caulk, since wood isn’t dimensionally stable compared to porcelain.

Yes, that’s a big piece of porcelain. Nominally 30″ x 60″. It’s 6.5mm thick. It has a PEI rating of 5 (the highest rating), so it’s unlikely that I’ll ever scratch it. If it’s good enough for commercial flooring, it’s good enough for my desktops. The design is a marble look, and I continue to be amazed at how far along this stuff has come in 10 years. It’s pretty, and zero maintenance (unlike real stone). Plus it’s manmade; it doesn’t involve carving up the planet. And I can move it by myself.

The downside… my desk base doesn’t meet the deflection requirements for tile or porcelain. That means that until the desk is in place, I will likely not install the insert. And I’m not going to use thinset to set it, since it’d likely just crack. I considered Ditra, but I don’t really trust it to hold the porcelain if I ever move the top later. While I love Ditra, I’ve never tried it in an inverted situation which might occur if two people are moving the top. Plus thinset would make the whole thing even heavier. Note that the porcelain alone is 38 pounds. I’m guessing that the whole top will be more than 80 pounds. My intent at the moment is to use Loctite PL Premium MAX to adhere the porcelain to the plywood. I may change my mind, but I think it’s my best option.

At any rate, I’ve installed the frame on the plywood base with TiteBond III and many cabinet screws from underneath. I also pin nailed and glued the trim pieces (not seen in picture above), which are clamped and curing.

Posted on

mcrover now monitors Plex server

mcrover is now monitoring my Plex server.
This was more work than expected. A big part of the issue here is that the REST API uses XML. I’ve always disliked using XML. It’s a nice technology, but when it comes to open source libraries for C++, it’s always been lacking.

Long ago, I used Xerces. Not because it’s the best, but because it was the only liberally licensed library with support for DTD and Schema validation. That is still the case today. Unfortunately, it’s very cumbersome to use and is written in old C++ (as in C++ 1998). There’s a lot of boilerplate, a considerable amount of global state (very bad for multithreaded applications), and a lot of the memory management is left to the application. I can’t imagine anyone today is using it in production.

But I plunged ahead anyway. Sadly, it was a mistake. Somewhere it was stomping on the stack, often in ways that caused problems deep inside openssl (which I don’t use directly, instead using boost::beast and boost::certify). The stack corruption caused problems trying to debug, and I didn’t have the time to figure it out. And of course I’m always suspicious of openssl, given the fact that it’s written in C and many of us lived through Heartbleed and many other critical openssl vulnerabilities. To be honest, we’ve been in desperate need of a really good, modern (and in at least C++11) C++ implementation of TLS for more than a decade. Of course I could rant about our whole TLS mess for hours, but I’ll spare you.

Time being of the essence, I switched to pugixml. I don’t need DTD/Schema validation. Problem gone, a lot less code, and a much more modern API (much harder to shoot yourself in the foot).

Inside mcrover, I’m using XPath on the XML returned by Plex. The internal code is generic; it would not be much work to support other Web applications with XML interfaces. The XPath I look for in the XML is a configuration item, and really the only reason I have something specific for Plex is their use of an API token. But the configuration is generic enough that supporting other XML Web applications shouldn’t be difficult.

At any rate, what I have now works. So now I don’t get blackholed fixing a Plex issue when I haven’t used it for months and something has gone wrong. I know ahead of time.

Posted on

UPS fiasco and mcrover to the rescue

I installed a new Eaton 5PX1500RT in my basement rack this week. I’d call it “planned, sort of…”. My last Powerware 5115 1U UPS went into an odd state which precipitated the new purchase. However, it was on my todo list to make this change.

I already own an Eaton 5PX1500RT, which I bought in 2019. I’ve been very happy with it. It’s in the basement rack, servicing a server, my gateway, ethernet switches and broadband modem. As is my desire, it is under 35% load.

The Powerware 5115 was servicing my storage server, and also under 35% load. This server has dual redundant 900W power supplies.

Installation of the new UPS… no big deal. install the ears, install the rack rails, rack the UPS.

Shut down the devices plugged into the old UPS, plug them in to the new UPS. Boot up, check each device.

Install the USB cable from the UPS to the computer that will monitor the state of the UPS. Install Network UPS Tools (nut) on that computer. Configure it, start it, check it.

This week, at this step things got… interesting.

I was monitoring the old Powerware 5115 from ‘ria’. ‘ria’ is a 1U SuperMicro server with a single Xeon E3-1270 V2. It has four 1G ethernet ports and a Mellanox 10G SFP+ card. Two USB ports. And a serial port which has been connected to the Powerware 5115 for… I don’t know, 8 years?

I can monitor the Eaton 5PX1500RT via a serial connection. However, USB is more modern, right? And the cables are less unwieldy (more wieldy). So I used the USB cable.

Trouble started here. The usbhid-ups driver did not reliably connect to the UPS. When it did, it took a long time (in excess of 5 seconds, an eternity in computing time). ‘ria’ is running FreeBSD 12.3-STABLE on bare metal.

I initially decided that I’d deal with it this weekend. Either go back to using a serial connection or try using a host other than ‘ria’. However…

I soon noticed long periods where mcrover was displaying alerts for many services on many hosts. Including alerts for local services, whose test traffic does not traverse the machine I touched (‘ria’). And big delays when using my web browser. Hmm…

Poking around, I seemed to only be able to reliably reproduce a network problem by pinging certain hosts with ICMPv4 from ria and observing periods where the round trip time would go from .05 milliseconds to 15 or 20 seconds. No packets lost, just periods with huge delays. These were all hosts on the same 10G ethernet network. ICMPv6 to the same hosts: no issues. Hmm…

I was eventually able to correlate (in my head) what I was seeing in the many mcrover alerts. On the surface, many didn’t involve ‘ria’. But under the hood they DO involve ‘ria’ simply because ‘ria’ is my primary name server. So, for example, tests that probe via both IPv6 and IPv4 might get the AAAA record but not the A record for the destination, or vice versa, or neither, or both. ‘ria’ is also the default route for these hosts. I honed in on the 10G ethernet interface on ‘ria’.

What did IPV4 versus IPv6 have to do with the problem? I don’t know without digging through kernel source. What was happening: essentially a network ‘pause’. Packets destined for ‘ria’ were not dropped, but queued for later delivery. As many as 20 seconds later! The solution? Unplug the USB cable for the UPS and kill usbhid-ups. In the FreeBSD kernel, is USB hoarding a lock shared with part of the network stack?

usbhid-ups works from another Supermicro server running the same version of FreeBSD. Different hardware (dual Xeon L5640). Same model of UPS with the same firmware.

This leads me to believe this isn’t really a lock issue. It’s more likely an interrupt routing issue. And I do remember that I had to add hw.acpi.sci.polarity="low" to /boot/loader.conf on ‘ria’ a while ago to avoid acpi0 interrupt storms (commented out recently with no observed consequence). What I don’t remember: what were all the issues I found that prompted me to add that line way back when?

Anyway… today’s lesson. Assume the last thing you changed has high probability of cause, even if there seems to be no sensible correlation. My experience this week: “Unplug the USB connection to the UPS and the 10G ethernet starts working again. Wait, what?!”.

And today’s thanks goes to mcrover. I might not have figured this out for considerably longer if I did not have alert information in my view. Being a comes-and-goes problem that only seemed to be reproducible between particular hosts using particular protocols might have made this a much more painful problem to troubleshoot without reliable status information on a dedicated display. Yes, it took some thinking and observing, and then some manual investigation and backtracking. But the whole time, I had a status display showing me what was observable. Nice!

Posted on

mcrover updates

I spent some weekend time updating mcrover this month. I had two motivations:

  1. I wanted deeper tests for two web applications: WordPress and piwigo. Plex will be on the list soon, I’m just not thrilled that their API is XML-based (Xerces is a significant dependency, and other XML libraries are not as robust or complete).
  2. I wanted a test for servers using libDwmCredence. Today that’s mcroverd and mcweatherd. dwmrdapd and mcblockd will soon be transitioned from libDwmAuth to libDwmCredence.

I created a web application alert, which allows me to create code to test web applications fairly easily and generically. Obviously, the test for a given application is application-specific. Applications with REST interfaces that emit JSON are pretty easy to add.

The alerts and code to check servers using libDwmCredence allow me to attempt connection and authentication to any service using Dwm::Credence::Peer. These are the steps that establish a connection; they are the prologue to application messaging. This allows me to check the status of a service using Dwm::Credence::Peer in lieu of a deeper (application-specific) check, so I can monitor new services very early in the deployment cycle.

Posted on

mcweather first pass completed

I’ve completed my first pass at ‘mcweather’, a caching weather server and command line client.

The main thrust here was to decouple weather condition and forecast fetching from qmcrover (a GUI client that is part of mcrover). I wanted a single server at home to fetch the weather information and provide it to local clients (command line and qmcrover). This avoids unnecessary traffic to the NWS (National Weather Service) web sites from local clients. This makes it faster as a consequence, since the local clients get the information from the local server.

It is secured using libDwmCredence. I changed qmcrover to use it, and in the process I updated two of my Raspberry Pis from buster to bullseye. One hurdle I encountered here: I was previously using a hand-built version of Qt 5.12. I didn’t want to continue doing so with bullseye, because Qt is very big and takes a long time to build. But the Raspberry Pi repositories don’t have two of the Qt libraries I need: libqt5webengine5 and libqt5webenginewidgets5. I hence added the debian repository to my list of apt repositories by adding it to /etc/apt/source.list:

deb http://deb.debian.org/debian bullseye main contrib non-free

I also created /etc/apt/preferences:

Package: *
Pin: origin raspbian.raspberrypi.org
Pin-Priority: 800

Package: *
Pin: origin deb.debian.org
Pin-Priority: 600

So far, so good. I’m running the server (mcweatherd) on the Raspberry Pi 4 that runs qmcrover in full screen mode in my office.

dwm@pi4e:/home/dwm% psg mcweatherd

USER   PID %CPU %MEM   VSZ  RSS TTY STAT START TIME COMMAND
root  8877  0.0  0.2 32832 8680 ?   Sl   01:10 0:02 /usr/local/sbin/mcweatherd