CI/CD For Firmware Development, Part I

Continuous Integration / Continuous Delivery, or more popularly known as CI/CD is probably one of the biggest hot-button topics in software development. There are hundreds, if not thousands of articles on the subject. That being said, I will not dive into the specifics of CI/CD itself. Instead I will (try) to put CI/CD in the context of embedded software development.

CI/CD at its core is essentially a set of development practices which ensure that quality software is delivered (and on time). The results of a CI/CD workflow, includes, but is not limited to – a considerably reduced development cycle, quick discovery and resolution of software bugs and overall increase in software quality.

To help put the concept into the context of embedded software development, ask yourself:

• Where is your code hosted?
• How do you integrate code from members of your team?
• How do you quickly ensure code quality?
• How do you quickly verify that new code does not inadvertently introduce bugs?
• How do you quickly ensure that new code does not break existing code/functionality?
• How do you test your code?
• How do you ship/release your code?

If you’re in the field, then chances are you can partially answer some of these questions. In some cases, you’re probably scratching your head, or even better, daydreaming about how great it would be to be able to do those things. If you fit in any of these response groups, trust me, you’re not alone.

Embedded software development is fun (or at least it can be under the right conditions), but more importantly, it’s hard – and that’s an understatement. Being a successful embedded software developer requires a certain mindset, and I believe that adopting CI/CD principles will go a long way – both for you and your team.

In the rest of this article, we’ll answer these questions by exploring 8 concrete steps to establishing a decent CI/CD flow for your project/team.

1. Version Control

Version control (VC) ensures that you have a full history of file changes in your project. Here, you do 3 things:

• Choose a VC tool. There are a number of version control tools, the most popular ones being Git, Subversion and CVS.

• When you’ve chosen your desired VC tool, the next step is to choose a hosting service. Here you have quite a lot of options – GitHub, GitLab, Bitbucket, just to name a few.

• Decide whether you want to host the code on the vendor’s cloud or on your own enterprise systems. Most (if not all) vendors give you the option of hosting your code on your own servers or on the cloud.

This might seem like a no-brainer, but you’ll be surprised to know there are still a lot of developers and teams that do not use VC. Come to think of it, I can’t possibly imagine how developers managed tons of code before version control was invented. It must have been a nightmare.

2. Develop Team Workflow And Branching Strategy

Team workflows and branching strategies vary, depending on the team and complexity of the project. In any case, I strongly believe these core workflows and strategies should be common to every team:

• Have an issue tracking system. Jira and ClickUp are some of my personal favorites.

• Each and every issue/task MUST be implemented on its own branch.

• For traceability, link issues to branches. In most cases, this already happens as a result of integration between your issue tracking solution and version control system. For example, consider Jira and Bitbucket. If you created an issue in Jira, you can create a Bitbucket branch with a single click. By default, the branch would be named after the issue.

• Code reviews MUST be conducted before merging branches. In the Git world, this is known as a “pull request”. Ensure that direct commits/merges to the master branch are disabled! This can easily be enforced using the administrator settings in your repository tool (GitLab/GitHub/Bitbucket, etc).

• Teams should decide and enforce the criteria for merging branches to the “master” branch (passing tests, code/test coverage). It is important the team exercises discipline, otherwise, it’s a disaster waiting to happen!

3. Setup CI/CD Server

There are many options, however my personal favorites in this category are Jenkins and GitLab, particularly because they are suitable for embedded development. They allow you to define physical machines (Jenkins calls them Slaves and Nodes, GitLab calls them Runners) where you can install tools to be used for compiling your code and executing hardware-in-loop automated tests. GitHub has also recently (at the time of writing this article) also introduced Runners, but the implementation is still fairly in its early stages.

Set up the master instance (either onsite or cloud), and then set up physical machines (nodes/runners). Ideally you can set up as many nodes/runners as possible. This would allow you to distribute build and test jobs. This would be especially beneficial in a scenario where you’re working with a fairly large/complex project and there’s a lot of tests to be executed. Ensure that tools installed on all machines are the same version.

4. Build Automation

As soon as your CI/CD server is up and running, the next step is to implement build scripts and integrate with your CI/CD server:

• Define build scripts (.sh or .bat) that compile your firmware. Ideally you create this script on your own development machine first and ensure that you can use this script to compile your application from the command prompt. The goal is to duplicate your development environment on your server. For example, if you use Segger Embedded Studio (SES) to write and build your code, then you should install the same tool (and same version) on your slave/runner. SES comes with emBuild, a utility that lets you compile your code from command-line. Most vendors these days have command-line utilities bundled with their software. Alternatively, if you are using the CMake system, then be sure to install the same on your slave/runner.

• Define pipeline scripts – these are scripts used by your CI/CD server to build code, run tests, deploy images, etc. Pipelines are split into stages. A stage performs a concrete action, such as building your code, or running unit tests. In most cases, building your code is the very first step in the pipeline. In Jenkins, the pipeline configuration is defined in a so-called JenkinsFile, and as for GitLab, you create a .gitlab-ci.yml file which defines the pipeline configuration.

• Automate! Configure your CI/CD server to automatically build whenever commits are made to branches. In GitLab this happens automatically; as for Jenkins, you need a bit of setup to get this to work.

At this point, we’ve established basic version control, workflows and build automation.
In Part II of this article, we’ll examine the testing aspects of CI/CD.