Continuous Integration / Continuous Delivery, or more popularly known as CI/CD is probably one of the biggest hot-button topics in software development. There are hundreds, if not thousands of articles on the subject. That being said, I will not dive into the specifics of CI/CD itself. Instead I will (try) to put CI/CD in the context of embedded software development. CI/CD at its core is essentially a set of development practices which ensure that quality software is delivered (and on time). The results of a CI/CD workflow, includes, but is not limited to - a considerably reduced development cycle, quick discovery and resolution of software bugs and overall increase in software quality.
To help put the concept into the context of embedded software development, ask yourself:
Where is your code hosted?
How do you integrate code from members of your team?
How do you quickly ensure code quality?
How do you quickly verify that new code does not inadvertently introduce bugs?
How do you quickly ensure that new code does not break existing code/functionality?
How do you test your code?
How do you ship/release your code?
If you’re in the field, then chances are you can partially answer some of these questions. In some cases, you’re probably scratching your head, or even better, daydreaming about how great it would be to be able to do those things. If you fit in any of these response groups, trust me, you’re not alone. Embedded software development is fun (or at least it can be under the right conditions), but more importantly, it’s hard - and that’s an understatement. Being a successful embedded software developer requires a certain mindset, and I believe that adopting CI/CD principles will go a long way - both for you and your team. In the rest of this article, we’ll answer these questions by exploring 8 concrete steps to establishing a decent CI/CD flow for your project/team.
Version control (VC) ensures that you have a full history of file changes in your project. Here, you do 3 things:
Choose a VC tool. There are a number of version control tools, the most popular ones being Git, Subversion and CVS.
When you’ve chosen your desired VC tool, the next step is to choose a hosting service. Here you have quite a lot of options - GitHub, GitLab, Bitbucket, just to name a few.
Decide whether you want to host the code on the vendor’s cloud or on your own enterprise systems. Most (if not all) vendors give you the option of hosting your code on your own servers or on the cloud.
This might seem like a no-brainer, but you’ll be surprised to know there are still a lot of developers and teams that do not use VC. Come to think of it, I can’t possibly imagine how developers managed tons of code before version control was invented. It must have been a nightmare.
Team workflows and branching strategies vary, depending on the team and complexity of the project. In any case, I strongly believe these core workflows and strategies should be common to every team:
Have an issue tracking system. Jira and ClickUp are some of my personal favorites.
Each and every issue/task MUST be implemented on its own branch.
For traceability, link issues to branches. In most cases, this already happens as a result of integration between your issue tracking solution and version control system. For example, consider Jira and Bitbucket. If you created an issue in Jira, you can create a Bitbucket branch with a single click. By default, the branch would be named after the issue.
Code reviews MUST be conducted before merging branches. In the Git world, this is known as a “pull request”. Ensure that direct commits/merges to the master branch are disabled! This can easily be enforced using the administrator settings in your repository tool (GitLab/GitHub/Bitbucket, etc).
Teams should decide and enforce the criteria for merging branches to the “master” branch (passing tests, code/test coverage). It is important the team exercises discipline, otherwise, it’s a disaster waiting to happen!
There are many options, however my personal favorites in this category are Jenkins and GitLab, particularly because they are suitable for embedded development. They allow you to define physical machines (Jenkins calls them Slaves and Nodes, GitLab calls them Runners) where you can install tools to be used for compiling your code and executing hardware-in-loop automated tests. GitHub has also recently (at the time of writing this article) also introduced Runners, but the implementation is still fairly in its early stages. Set up the master instance (either onsite or cloud), and then set up physical machines (slaves/runners). Ideally you can set up as many slaves/runners as possible. This would allow you to distribute build and test jobs. This would be especially beneficial in a scenario where you’re working with a fairly large/complex project and there’s a lot of tests to be executed. Ensure that tools installed on all machines are the same version.
As soon as your CI/CD server is up and running, the next step is to implement build scripts and integrate with your CI/CD server:
Define build scripts (.sh or .bat) that compile your firmware. Ideally you create this script on your own development machine first and ensure that you can use this script to compile your application from the command prompt. The goal is to duplicate your development environment on your server. For example, if you use Segger Embedded Studio (SES) to write and build your code, then you should install the same tool (and same version) on your slave/runner. SES comes with emBuild, a utility that lets you compile your code from command-line. Most vendors these days have command-line utilities bundled with their software. Alternatively, if you are using the CMake system, then be sure to install the same on your slave/runner.
Define pipeline scripts - these are scripts used by your CI/CD server to build code, run tests, deploy images, etc. Pipelines are split into stages. A stage performs a concrete action, such as building your code, or running unit tests. In most cases, building your code is the very first step in the pipeline. In Jenkins, the pipeline configuration is defined in a so-called JenkinsFile, and as for GitLab, you create a .gitlab-ci.yml file which defines the pipeline configuration.
Automate! Configure your CI/CD server to automatically build whenever commits are made to branches. In GitLab this happens automatically; as for Jenkins, you need a bit of setup to get this to work.
Now that build scripts and pipeline scripts have been setup and running on your CI server, the stage is set. From here, you expand your pipeline and add more stages. For the next step, I’d recommend adding Static Code Analysis (SCA). Static code analysis scans your code to reveal potential vulnerabilities such as null pointer problems, buffer overflows, memory leaks, division by zero, etc. Upon completion of scanning your code, the SCA tool generates a report showing the vulnerable areas of our code, as well as the severity levels of the detected issues. Two things to keep in mind when using SCA tools:
I often find that different SCA tools have their strengths and weaknesses, so I’d advise to combine at least 2 different tools. for example, Flawfinder is very good at catching potential buffer overflow errors and format string errors, but does not consider data type issues. SPLINT on the other hand, does much deeper analysis.
The code analysis sometimes produces false negatives; not every reported issue is necessarily a problem. You’ll have to asses each issue in the given context.
Depending on your team ambition level (and budget), you might want to opt for open-source or proprietary solutions. Personally I don’t have a specific preference when it comes to SCA tools; I combine 2-3 different tools, and that’s more than enough.
Once again, this is a step that can be automated. Create scripts that run static code analysis and add them to your CI pipeline. If you want to be strict (and I’d recommend a reasonable level of strictness), you can set minimum severity levels that will let your pipeline fail. This will enable the team to quickly act and fix the error. For example, Flawfinder uses a scale of 0 - 5 to define vulnerability levels. You could then define your script to fail the analysis if a vulnerability of level 3 or above is detected. That should keep your team on their toes.
Unit testing requires a bit more work to setup compared to SAST. If you are a fan of TDD (Test Driven Development), then this is for you. Personally, I never really got into the TDD mindset - which essentially requires you to write tests before you write the actual code (or at least write code and tests simultaneously).
If your teams happens to be one of the few that practice TDD, I say, weldone. And if you’re looking to get into TDD, look no further than James W. Greening’s Test Driven Development for Embedded C. You can also automate unit test runs by adding the relevant scripts to your CI pipeline. You could use the CppUTest framework to write your unit tests.
My favorite type of testing. Essentially HIL testing, at least in the context of embedded development involves executing tests on the physical embedded device itself. Conceptually, here are the key elements involved:
A test script - written in Python, C++ or any preferred language. The script runs on your CI slave/runner.
The embedded device is physically connected to the CI slave/runner. Depending on your embedded device, it may be powered via the CI slave/runner, or a separate power source. You might need to have control mechanisms (like relays, for example) that provide external stimuli to your embedded device.
Through the test script, commands are sent to the embedded device.
A bi-directional transport protocol (like UART), implemented between your script and the target embedded device, allows for the script to communicate with the embedded device, and vice versa.
The embedded device receives a command from the script and executes the relevant subroutines. You would have to implement a module in your embedded code which would specifically handle execution of these test commands and sending back responses via the implemented transport protocol. I’d strongly recommend you to try as much as possible to decouple this module from the rest of your application code; remember it’s only purpose is for testing, and will not be in the final release.
The embedded device sends a response back to the script. The script validates the response, and the next command is sent, and so on.
There are a number of test frameworks you can use for embedded HIL testing - I’d highly recommend Pytest or Robot Framework. And once more, not to sound like a broken record, automate! Add the scripts to your CI pipeline so that tests will run on every commit. You can split your tests into groups (or suites), in such a way that quick, sanity tests will run whenever daily commits are made. Then you can configure your pipeline to run nightly, where ALL tests, which could potentially take hours, will be executed. Another way of splitting your tests is to group them by feature. The options are limitless.
If you’re visual like me, you’ll also setup a TV dashboard so you can see build and test results when you come to work in the morning, as well as monitor build and test activities during the day. I’d warn you though, coming to the office in the morning to see a dashboard where all tests have failed could be a stressful way to start your day. It’s never a great start to come in at 9 in the morning and the first thing you say is the F word.
Your release workflow will also heavily depend on your team, project requirements, and your target customers (either another team in the company or end users of your software). In any case, find a workflow that works best for you, and, you guessed it.. automate!
Automate your release workflow as much as you can. Your CI server is there to make it happen. Use it to create tags, release candidates, whatever suits you. Finally, be sure to have a versioning strategy for your software. Semantic versioning tends to work well in most cases.
At the very minimum, you should have the following pipeline stages running on your CI server everyday (multiple times a day):
Build
Static Code Analysis
Unit tests and/or HIL tests
Setting all this up is the hard part. Once the mechanisms are in place, it’s a matter of maintenance. You’ll have to do a lot of scripting to get this workflow running because every embedded project is unique; we don’t have the luxury of pulling generic tools off the shelf.
Automate as much as you can, wherever and whenever it makes sense to do so. Not everything can be automated unfortunately; do not underestimate the important of manual testing. Do not fully rely on automated tests.
In an ideal situation, you want to establish this workflow as soon as possible when you start a new project. Then it becomes a daily routine for your team. As for already existing projects, your team will have to invest considerable time, resources and effort. But trust me, it will be time well and truly spent. In the end, it’s all worth it, and it gives you and your team peace of mind.