In Part I of this article, we looked at how to “kickstart” the CI/CD mindset during your development process. In this article, we round things off by examining vital component of this process – testing.

5. Test Automation – Static Code Analysis

Now that build scripts and pipeline scripts have been setup and running on your CI server, the stage is set. From here, you expand your pipeline and add more stages. For the next step, I’d recommend adding Static Code Analysis (SCA). Static code analysis scans your code to reveal potential vulnerabilities such as null pointer problems, buffer overflows, memory leaks, division by zero, etc. Upon completion of scanning your code, the SCA tool generates a report showing the vulnerable areas of our code, as well as the severity levels of the detected issues.
Two things to keep in mind when using SCA tools:

• I often find that different SCA tools have their strengths and weaknesses, so I’d recommend to combine at least 2 different tools. for example, Flawfinder is very good at catching potential buffer overflow errors and format string errors, but does not consider data type issues. SPLINT on the other hand, does much deeper analysis.

• The code analysis sometimes produces false negatives; not every reported issue is necessarily a problem. You’ll have to asses each issue in the given context.

Depending on your team ambition level (and budget), you might want to opt for open-source or proprietary solutions. Personally I don’t have a specific preference when it comes to SCA tools; I combine 2-3 different tools, and that’s more than enough.

Once again, this is a step that can be automated. Create scripts that run static code analysis and add them to your CI pipeline. If you want to be strict (and I’d recommend a reasonable level of strictness), you can set minimum severity levels that will let your pipeline fail. This will enable the team to quickly act and fix the error. For example, Flawfinder uses a scale of 0 – 5 to define vulnerability levels. You could then define your script to fail the analysis if a vulnerability of level 3 or above is detected. That should keep your team on their toes.

6. Test Automation  – Unit Testing

Unit testing requires a bit more work to setup compared to SAST. If you are a fan of TDD (Test Driven Development), then this is for you. Personally, I never really got into the TDD mindset – which essentially requires you to write tests before you write the actual code (or at least write code and tests simultaneously).

If your teams happens to be one of the few that practice TDD, I say, weldone. And if you’re looking to get into TDD, look no further than James W. Greening’s Test Driven Development for Embedded C.
You can also automate unit test runs by adding the relevant scripts to your CI pipeline. You could use the CppUTest framework to write your unit tests.

7. Test Automation  – Hardware-In-Loop (HIL) Testing

My favorite type of testing. Essentially HIL testing, at least in the context of embedded development involves executing tests on the physical embedded device itself. Conceptually, here are the key elements involved:

• A test script – written in Python, C++ or any preferred language. The script runs on your CI slave/runner.

• The embedded device is physically connected to the CI slave/runner. Depending on your embedded device, it may be powered via the CI slave/runner, or a separate power source. You might need to have control mechanisms (like relays, for example) that provide external stimuli to your embedded device.

• Through the test script, commands are sent to the embedded device.

• A bi-directional transport protocol (like UART), implemented between your script and the target embedded device, allows for the script to communicate with the embedded device, and vice versa.

• The embedded device receives a command from the script and executes the relevant subroutines. You would have to implement a module in your embedded code which would specifically handle execution of these test commands and sending back responses via the implemented transport protocol. I’d strongly recommend you to try as much as possible to decouple this module from the rest of your application code; remember it’s only purpose is for testing, and will not be in the final release.

• The embedded device sends a response back to the script. The script validates the response, and the next command is sent, and so on.

There are a number of test frameworks you can use for embedded HIL testing – I’d highly recommend Pytest or Robot Framework. And once more, not to sound like a broken record, automate! Add the scripts to your CI pipeline so that tests will run on every commit.

You can split your tests into groups (or suites), in such a way that quick, sanity tests will run whenever daily commits are made. Then you can configure your pipeline to run nightly, where ALL tests, which could potentially take hours, will be executed. Another way of splitting your tests is to group them by feature. The options are limitless.

If you’re visual like me, you’ll also setup a TV dashboard so you can see build and test results when you come to work in the morning, as well as monitor build and test activities during the day. I’d warn you though, coming to the office in the morning to see a dashboard where all tests have failed could be a stressful way to start your day. It’s never a great start to come in at 9 in the morning and the first thing you say is the F word.

8. Setup Release Workflow

Your release workflow will also heavily depend on your team, project requirements, and your target customers (either another team in the company or end users of your software). In any case, find a workflow that works best for you, and, you guessed it.. automate!

Automate your release workflow as much as you can. Your CI server is there to make it happen. Use it to create tags, release candidates, whatever suits you. Finally, be sure to have a versioning strategy for your software. Semantic versioning tends to work well in most cases.

Conclusion

At the very minimum, you should have the following pipeline stages running on your CI server everyday (multiple times a day):

• Build
• Static Code Analysis
• Unit tests and/or HIL tests

Setting all this up is the hard part. Once the mechanisms are in place, it’s a matter of maintenance. You’ll have to do a lot of scripting to get this workflow running because every embedded project is unique; we don’t have the luxury of pulling generic tools off the shelf.

Automate as much as you can, wherever and whenever it makes sense to do so. Not everything can be automated unfortunately; do not underestimate the important of manual testing. Do not fully rely on automated tests.

In an ideal situation, you want to establish this workflow as soon as possible when you start a new project. Then it becomes a daily routine for your team. As for already existing projects, your team will have to invest considerable time, resources and effort. But trust me, it will be time well and truly spent.

In the end, it’s all worth it, and it gives you and your team peace of mind.