CI/CD for Embedded Software Development

Overview

Continuous Integration / Continuous Delivery (CI/CD) is a set of development practices that ensure quality software is delivered on time. The results include a considerably reduced development cycle, quick discovery and resolution of bugs, and an overall increase in software quality.

To put the concept in the context of embedded software development, ask yourself:

• Where is your code hosted?
• How do you integrate code from members of your team?
• How do you quickly ensure code quality?
• How do you quickly verify that new code does not inadvertently introduce bugs?
• How do you test your code?
• How do you ship/release your code?

In the rest of this article, we’ll explore 8 concrete steps to establishing a solid CI/CD flow for your embedded project.

Version Control

Version control ensures that you have a full history of file changes in your project. Three things to do:

• Choose a VC tool. The most popular ones are Git, Subversion, and CVS.
• Choose a hosting service: GitHub, GitLab, Bitbucket, and others.
• Decide whether you want to host the code on the vendor’s cloud or on your own enterprise systems.

This might seem like a no-brainer, but you’ll be surprised how many developers and teams still don’t use version control.

Develop Team Workflow and Branching Strategy

Core workflows and strategies that should be common to every team:

• Have an issue tracking system. Jira and ClickUp are personal favorites.
• Each and every issue/task must be implemented on its own branch.
• For traceability, link issues to branches.
• Code reviews must be conducted before merging branches. Ensure that direct commits/merges to the master branch are disabled.
• Teams should decide and enforce the criteria for merging branches (passing tests, code coverage). It is important the team exercises discipline, otherwise it’s a disaster waiting to happen.

Setup CI/CD Server

My personal favorites in this category are Jenkins and GitLab, particularly because they are suitable for embedded development. They allow you to define physical machines (Jenkins calls them Slaves and Nodes, GitLab calls them Runners) where you can install tools for compiling code and executing hardware-in-loop automated tests.

Set up the master instance (either onsite or cloud), then set up physical machines. Ideally set up as many slaves/runners as possible to distribute build and test jobs. Ensure that tools installed on all machines are the same version.

Build Automation

Once your CI/CD server is up and running:

• Define build scripts (.sh or .bat) that compile your firmware. Create and test the script on your own development machine first, then duplicate the environment on your server.
• Define pipeline scripts used by your CI/CD server to build code, run tests, and deploy images. In Jenkins, this is defined in a JenkinsFile; for GitLab, you create a .gitlab-ci.yml file.
• Configure your CI/CD server to automatically build whenever commits are made to branches.

Static Code Analysis

Static code analysis (SCA) scans your code to reveal potential vulnerabilities such as null pointer problems, buffer overflows, memory leaks, and division by zero.

Two things to keep in mind:

• Different SCA tools have their strengths and weaknesses. Combine at least 2 different tools. For example, Flawfinder is very good at catching potential buffer overflow errors, while SPLINT does much deeper analysis.
• The code analysis sometimes produces false negatives. You’ll have to assess each issue in context.

Create scripts that run static code analysis and add them to your CI pipeline. If you want to be strict (recommended), set minimum severity levels that will cause your pipeline to fail.

Unit Testing

Unit testing requires more setup than SAST. If you are a fan of TDD (Test Driven Development), this is for you. You can automate unit test runs by adding the relevant scripts to your CI pipeline. The CppUTest framework works well for writing unit tests in C.

Hardware-In-Loop Testing

My favorite type of testing. HIL testing involves executing tests on the physical embedded device itself. Key elements:

• A test script written in Python, C++, or any preferred language, running on your CI slave/runner.
• The embedded device is physically connected to the CI slave/runner.
• Commands are sent to the embedded device through the test script via a bi-directional transport protocol (like UART).
• The embedded device receives a command, executes the relevant subroutines, and sends a response back.
• The script validates the response, sends the next command, and so on.

Recommended test frameworks: Pytest or Robot Framework.

Split your tests into groups: quick sanity tests run on every daily commit; nightly runs execute ALL tests. You can also group tests by feature.

Setup Release Workflow

Your release workflow will heavily depend on your team, project requirements, and target customers. Find a workflow that works best for you, and automate as much as possible. Use your CI server to create tags, release candidates, and whatever suits you. Adopt a versioning strategy for your software. Semantic versioning tends to work well in most cases.

Conclusion

At the very minimum, you should have the following pipeline stages running on your CI server every day (multiple times a day):

• Build
• Static Code Analysis
• Unit tests and/or HIL tests

Setting all this up is the hard part. Once the mechanisms are in place, it’s a matter of maintenance. Automate as much as you can, wherever and whenever it makes sense. Not everything can be automated; do not underestimate the importance of manual testing.

In an ideal situation, establish this workflow as early as possible when you start a new project. Then it becomes a daily routine. For existing projects, your team will have to invest considerable time, resources, and effort. But trust me, it will be time well and truly spent.