Quality Code Starts with Smarter Testing

In any kind of software development, but especially embedded, thoroughly testing your software is non-negotiable. Testing creates code that is reliable, maintainable, and capable of handling real-world challenges. It also prevents costly bugs and allows you to iterate with confidence. In this blog, I’ll share some key testing strategies and best practices, pulled from a whitepaper I recently wrote about general embedded development.

The foundation for reliable code

Testing needs to be baked into the development process – not an afterthought. That means you should have a robust suite of unit tests at the heart your software. Some frameworks you might want to look at are Google Test/gMock, doctest, Catch2, and Qt Test.

Many developers see testing as tedious or boring, but this mindset needs to change. Unit tests act as a safety net, letting you tweak and improve code without worrying that you’re going to break something. They are one of the best tools you have to ensure your code is dependable when things go wrong.

Comprehensive code coverage is key. During code reviews, flag any untested sections and take the time to flesh out those tests before committing your other changes. Similarly, bug fixes should be considered an opportunity to strengthen the test suite by writing tests that recreate the problem.

Speed also matters. Tests that run in seconds are much more likely to be used frequently, enabling developers to catch issues early. Tests that take too long often aren’t used during daily development. Longer-running tests are fine for CI pipelines, but your day-to-day tests should be quick and easy to deploy.

Here are a couple recommendations for letting tests run quickly in your daily test runs:

• Keep input data small: You can always use more comprehensive data sets available for overnight runs.
• Leave out benchmarks: Those can always be done on demand and aren’t typically needed when you’re just quickly looking to see if you’ve broken anything.
• Run tests in parallel: This is a capability that can be automatically built into the test framework. (For example, if you’re using the CTest framework in CMake, you can do this by launching ctest with the “-j” option or equivalently “--parallel”, as well as through setting the CTEST_PARALLEL_LEVEL environment variable.) The tests can’t interfere with each other if you want to successfully run them in parallel. In other words, don’t modify shared resources like files, and instead moving those operations to use in-memory copies or instance-specific temporary directories.

HAL and HIL

When testing embedded systems, it’s important to consider how to test hardware-related features. By separating hardware calls into a hardware abstraction layer (HAL,) you can isolate hardware-related functionality, making it easier to stub it out for testing purposes. Depending on your needs, there are several possible testing options – and often a mix of them works best:

• Hardware-in-loop (HIL): This method uses real hardware rigs to test the system, offering high test coverage and system stability since tests run on the intended hardware. However, HIL is labor-intensive and costly, requiring manual updates and significant investment in maintaining the hardware rigs.
• Virtualized hardware: With this approach, you write software to emulate the more critical portions of hardware functionality. The goal isn’t to mimic every hardware state but to emulate enough functionality so it behaves like the real hardware. This isn’t going to find hardware-dependent bugs, but it allows you test functions that depend on hardware more comprehensively.
• No hardware testing: In cases where hardware is too expensive, scarce, or difficult to emulate, you may need to skip testing certain hardware-dependant functionality. While not ideal, this may be the only option in some situations. If you must forgo hardware testing, make sure to document the trade-offs and plan for additional testing on actual hardware as part of system validation.

The data-driven advantage

Writing and maintaining separate tests for every edge case can quickly get out of hand. That’s where data-driven testing comes in. Instead of creating specialized test code for each case, developers feed a variety of inputs and expected outputs from external sources, like files or spreadsheets, into the same test logic. Decoupling this logic from test data is a smarter, more efficient way to handle complex testing. Advantages include:

• Scalability: Adding new cases is as simple as updating a data file or spreadsheet, so teams can cover more scenarios with less effort. Plus, compared to editing and recompiling C++ code, data files can be easily edited. That makes it possible for testers to add or fix tests, not just developers, which can help spread out the burden of maintaining a large test suite.
• Versatility: Data-driven testing works particularly well for parsers and other code that processes or interprets data, but it’s also applicable to a wide range of use cases.
• Real-world accuracy: Logs or execution outputs can be repurposed as input scripts to replicate real-world behaviors that are hard to simulate manually.

With data-driven techniques, you can cover more ground without creating unnecessary headaches. It’s a practical, efficient way to ensure even edge cases are accounted for.

Crash reporting kept simple

No matter how much testing you do, crashes happen – especially in embedded systems running in the field. Unlike mobile platforms that provide built-in crash reporting, embedded developers need to roll their own solutions.

The good news is, a practical crash reporting system doesn’t need to be complicated. Minidumps, for example, are a lightweight alternative to full core dumps. These smaller files capture key data, such as registers, stack frames and the instruction pointer, without bloating files or collecting sensitive data. Pairing minidumps with logs and build version information lets you pinpoint issues quickly.

To make the most of crash reports, use a back-end system to collect and process them. Tools like Google Breakpad and Sentry.io can automate much of this process, linking crash data with the correct build symbols and grouping similar issues for easier diagnosis. While more advanced tools like Crashpad are available, simpler systems like Breakpad often provide an ideal balance of functionality and ease of integration. (Sometimes a running start with a straightforward tool is better than being bogged down by the complexities of advanced systems.)

Insights from in-field reporting

Remember, not all bugs are catastrophic crashes. Many issues are subtle malfunctions that don’t immediately crash the software but still result in a poor UX. These bugs often show up in the field where end users are pushing your code in unexpected ways. Empowering users to report these issues is just as important as capturing crash data.

A robust in-field reporting system should collect all the information you need for understanding the problem, such as thread stacks, logs, and any contextual data. Combine this with a customer’s description of the problem, and you’ll have a clear view of the issue, making it easier to reproduce and resolve. In particular, logs are super helpful because they show you what was happening when the software didn’t work as expected.

Encouraging customers to report bugs also helps builds trust. When users see that you’re listening and acting on their feedback, they’re more likely to remain loyal – even in the face of occasional software glitches.

Testing at the core of quality

Testing is not just a duty – it’s a mindset that separates professional software from mediocre solutions. By embracing these best practices, you’ll be setting yourself up for success, ensuring that your software not only performs reliably but also stands up to the challenges of real-world use.

Want more insights into best practices for embedded software development? Download my full whitepaper for more info on mastering embedded engineering.