Not too long ago, we had an issue on my team that blackholed over a week of my time.
Spoiler alert: the overarching lesson here is to always question and improve your processes, lest they bite you.
I’m currently working on an embedded product with a small amount of safety requirements. As such, the target software must check the validity of the contents of ROM (containing code and constant data) on an ongoing basis. It also must check the stack for corruption or growth above a predetermined bound, run processor tests, etc. The response to problems discovered in any of these tests is a reset followed by a “safe state”.
Fine. Not rocket science, and very common. Easy, right? Well, yes, if your development process is checked for sanity…
A development detail: our chosen method of proving ROM validity is to add a 32-bit CRC to the end of each section of ROM that must be checked for validity. Each CRC is calculated at build time, and needs to wind up in ROM on the target microcontroller so that the software on the target can recompute the CRC at runtime on a periodic basis and compare its result to the precalculated CRC. If the value calculated at runtime doesn’t match the precalculated value stored in ROM, the ROM is considered invalid and remediation must occur. In our case, a reset followed by a “safe state”.
Problem 1: “How do I get that precalculated CRC into the ROM image for the target as part of the build process?” The easy answer is to insert it into the S-Record file that is used as input to the flash programmer or debugger when flashing the target. It’s easy because S-Record files are trivial to parse and many tools are readily available for this step of the process (Vector’s Hexview, the open source srec, et. al.). In our case, we’re using my own dwmhex tool that I wrote many years ago.
Problem 2: here’s where previous project ran into trouble. The debugger needs an ELF file when debugging. The S-Record file is absent of symbols and other information needed for productive debugging. We don’t modify the ELF file to add the CRC information (though we could, it’s been too long since I’ve used libelf for it to be a productive use of my time).
There must be a way to tell the debugger to load symbols and other debugging information from the ELF file but to flash the target with the S-Record file, right? Yes, of course. Any embedded debugger worth its salt provides this functionality.
Unfortunately, for some reason, previous projects were not aware of how to leverage this common debugger feature. Hence they always loaded only the ELF file in the debugger, and let the debugger translate the ELF when flashing the target. Big problem: the ELF doesn’t contain the precalculated CRC values, so if you let the code run, it will result in a reset. Quickly. Making debugging impossible.
The solution in previous projects? Preprocessor conditionals preventing the inclusion of the ROM CRC calculation and other test code in debug builds. This is terrible practice. A rough analogy: if you built a bridge, then drove an empty truck across it, would you then declare it good enough to handle a fully loaded truck? You can’t debug code that wasn’t even compiled into the image for the target. And as a general rule, you should avoid producing dramatically different code for release versus debugging when the debugger doesn’t work with release builds.
How we were bitten…
One of our young engineers was thrown into the project without proper guidance. He added the sanity-checking code, but he only tested using the debugger and a “debug” build. Because he had wrapped all of the sanity-checking code with preprocessor conditionals that excluded the sanity-checking code from “debug” builds (as done on previous projects), most of the code he had just added was completely untested; it didn’t even exist in the “debug” build. He committed the code, not knowing whether or not it actually worked for release builds. And as other team members were busy with other tasks (it was a development release week), no one noticed until about a week later when everyone updated their working copies, compiled a “release” build, and flashed their devices. And in the process, turned them into bricks. The resets were so frequent that it wasn’t even possible to reflash the parts via the bootloader. The bootloader would jump to the application and the application would reset nearly instantaneously.
Other team members reverted to earlier code so they could continue their work. I rolled up my sleeves and dug in my heels…
In the end, there were six different causes of resets in the code. It took me over a week to find and fix all of them. Our young engineer had assumed that we were already putting the CRC values into the S-Record files, which wasn’t true (we hadn’t added that to our build process yet). A set of assembly routines obtained from the microcontroller vendor for processor tests were committed without review, and unfortunately that code made some very egregious assumptions about how it could use RAM: it had hardcoded addresses for its RAM usage that caused it to stomp on our stack sentinel and we hence failed our stack test. etc.
But the bigger failure here was the violation of some fairly universal software tenets:
- Don’t create unnecessary differences between “debug” and “release” builds. It will waste precious development resources today or tomorrow, guaranteed.
- Unless prohibitively expensive to do otherwise, NEVER create release builds that have MORE code than debug builds. Or more accurately, don’t put code into a release build that isn’t readily debugged. Obviously, clicking on a few checkboxes in the debugger GUI does not qualify as prohibitively expensive, especially when it remembers your settings across debugging sessions (I’d call this essentially “free”).
- Trust and nurture your gut instinct. If you think, “There MUST be an easier way to do this…”, odds are good that you’re right. If the cost of not finding that way is anything above piddly, it’s your job as a software engineer to find that way. And it might be as easy as asking an engineer in the next aisle or blasting out an email. A specific example documented here: “There MUST be a way to tell the debugger to flash an S-Record file to the target while loading symbols and other debugging information from an ELF file!” You’re correct. Common problem with common solutions.
- Always question your processes. If something smells awful, it’s probably a turd. And as a cynical friend of mine likes to say, a polished turd is still a turd.
- Don’t throw your young engineers into a hot frying pan and then proceed to burn them to a crisp. Help them be successful!