I’ve been surprised over the years that I haven’t been able to find more documentation on techniques for improving the security of the development supply chain. Most practices seem to revolve around perimeter security (firewalls or self-hosted runners) or code scanning. While these can be an important part of the security process, the security of the supply chain is more often controlled by the processes for reviewing new suppliers.
What do I mean by that? In short, if you don’t have a strong process for reviewing your dependencies and supply chains, then you will always struggle to maintain a secure environment. Having a strong process for reviewing and approving dependencies is the start of a strong security foundation. If you’re planning to add to your supply chain, it’s important to understand the component and its risks. To do that, it helps to have a clear process.
Trusting the marketplace
Let’s consider a GitHub Action Are you relying on the verified badge from the GitHub Marketplace to determine if an Action is safe? You may be surprised to understand that this badge is not a security badge. It is a badge that indicates that the provide code meets the requirements for being listed. If the owner has applied for publisher verification and been verified, then it indicates that the publisher’s domain and an email address were verified. The article About marketplace badges provides more details.
In short, it is not an indication of anything related to the security of the code. This situation is similar with most package managers as well. There are limited checks to ensure the code is safe to use or meets your needs. The marketplace, repository, or registry exists to make you aware of options, not to ensure the quality of those offerings.
Defining the process
The process for trying is generally similar across all dependency types. There’s a few key steps to consider, and there are many opportunities for some level of automation. There are eight aspects to consider as you start to build a process:
- The vendor
- The license
- The code
- The quality
- The functionality
- The security posture
- The dependencies
- The findings
Let’s examine each of these and how to consider them in the context of a GitHub Action. We’ll assume we want to use the
actions/setup-node Action and need to review it before approving it for use in the organization.
Review the vendor
GitHub is a known software provider and has a business need to remain secure. Because the code is widely used, it’s also constantly being reviewed by the community. You can research the company and its security practices to understand how the code is built and the products that are provided. By comparison, smaller vendors may not have the same level of recognition. This can make it harder to know the practices behind the code. It can also mean that the code is not as frequently reviewed for security exploits by the community. This is not to say that you should not use smaller vendors, but that you should be aware of the risks and take steps to mitigate them.
Review the license
The first step in reviewing the Actions is to make sure that the license being provided is acceptable to the company. Because the licenses used can have legal impacts, it is always important to understand. That said, most Actions have their own dependencies. In that case, you will want to understand and review those as well. In the case of actions/setup-node, the license is the MIT license. This is a very permissive license and is likely to be acceptable to most companies. The license is provided in the LICENSE file in the repository. Because GitHub repositories can provide a list of dependencies, you should review those as well. This is found under the
Insights tab of the repository in the Dependency Graph blade:
This page also provides a button for exporting the software bill of materials (SBOM), allowing you to easily capture, review, and preserve the list of dependencies and their licenses.
This particular Action goes a step further. There is a file in the repository,
.licensed.yml, which indicates the allowed licenses. The repository relies on a
reusable workflow that uses the licensed tool to review the dependency chain during the PR process to ensure that no dependencies with unapproved licenses are merged into the repository. This is a great example of how to ensure that the license of the repository is maintained and how to ensure that unexpected licenses are not incorporated into the code.
Review the code
Looking at the
action.yml file, you can see what code will be executed and how. In this case, it will use Node.js version 20, calling the file dist/setup/index.js to run the Action itself.
Reviewing the file dist/setup/index.js in the repository is challenging. It’s a minimized file that contains all of the various dependencies it relies on. Is this the same code as what is in the src folder? Is it using the same packages declared in the packages.json and reported by the SBOM? It’s hard to know by looking at the code. In my mind, it’s important to follow the old security adage – trust but verify.
To assist with this, the repository relies on another reusable workflow,
check-dist.yml. With each merge or pull request, it rebuilds the code to confirm that the contents of the dist folder will match the compiled code. This ensures that the code remains consistent with the source code that is being provided. You can run these checks yourself, but in this case the vendor is ensuring that there will be frequent workflow runs that show the code remains consistent. I can verify the state rather than assuming. If I don’t trust the vendor, I can always run the checks myself.
Review the quality
The quality of the code is often a reflection of the quality of the process. A well-maintained codebase with well defined processes is more likely to be secure and reliable, and the code is more likely to be maintainable over time. The best of these practices will also allow you to validate the quality of the code and its behaviors yourself.
If we look at the Action’s code, it has automation to build and test the code (the shared workflow,
basic-validation.yml). This ensures the code can be built and that it has unit tests to verify behaviors. In addition, having this process available for pull requests means that new versions of its dependencies can be quickly and easily validated for regression issues.
The repository also contains prettierrc.js and eslintrc.js. These tools ensure that the code is linted and formatted. This provides consistent code standards, making the code easier to read and maintain. It also continuously notifies developers of any best practices that should be followed while coding. In addition, the basic validation workflow runs the linters to ensure any code being merged follows those standards and is properly formatted. If not, then it prevents the pull request from being merged until the issues are resolved.
One final thing to observe. Because the code predates Dependabot, it invokes native functionality in NPM to audit the packages (
npm audit) to look for security issues. This ensures that the code is not relying on packages with known security vulnerabilities. If a vulnerability is detected, the command line call returns a non-zero error code, breaking the workflow. In this case, the command line
npm audit --audit-level=high only breaks the build on high or critical vulnerabilities. Personally, I prefer to break the build on any vulnerabilities. As a result, I might decide to have automation to periodically check the code for vulnerabilities. If one is detected, I could automatically disallow the Action in the organization and I could raise an internal issue to review the Action.
Just like before, you can run all of these checks yourself. Seeing that the vendor has automated these checks and made them part of the development process provides greater confidence in the quality and maintainability of the code. In addition, it makes it easier for the community to contribute to the code.
Review the functionality
While we know the code should work for installing Node.js in a runner and we see that the unit tests are passing, do we know that this Action ultimately does what is required of it when it is used in a workflow? Does it setup node and prepare the cache?
The repository provides a workflow,
e2e-cache.yml which tests the Action by invoking it, then calling a
Bash script that validates the results. To do this, it relies on the fact that a workflow can always reference an Action from a local path. The workflow file references the Action with
uses: ./ to test that the current version of the code with each pull request or push to main.
It’s very easy with tests to say that the code should do some task. It’s another thing entirely to run tests that ensure that this assumption is true and accurate. Once again, we see this integrated into a development process, ensuring that mistakes are caught early and that the code remains functional. It also allows users to review the results and confirm the functionality (as well as test it themselves).
Review the security posture
We’ve seen how the code is reviewed for quality and how its supply chain is reviewed. The next thing to consider is coding mistakes that create vulnerabilities. Since this repository is in GitHub, I would expect to see the use of at least CodeQL to try to avoid mistakes that introduce vulnerabilities. The repository does not disappoint. It has a workflow that relies on the reusable workflow,
codeql-analysis.yml to analyze every pull request and periodically rescan the repo, in case a new vulnerability is discovered. This ensures that the code isn’t creating vulnerabilities that might impact its users.
It’s important to understand the security posture of the vendor and any steps being taken to avoid mistakes. If there are none, then it may mean that they react to security vulnerabilities rather than proactively prevent them. For other types of dependencies, I like to see additional scans. There’s a lot of security tooling available to help developers avoid mistakes. If they are not using them, then I may need to think carefully about trusting the code. Alternatively, I can run the checks myself (just like with the earlier NPM audit). Just make sure that it’s automated!
Review the dependencies
As a final consideration, I like to review the dependencies similarly. There’s a common expression – you don’t marry a person, you marry a family. The same is true with software. You aren’t taking a single dependency, you’re taking all of the dependencies it brings with it. In this case, there’s a large number of NPM packages that are being incorporated in the final product. We’ve seen some steps taken already to ensure the integrity and licensing on these components.
If this was a Dockerfile or Docker-based Action, then I may need additional steps to validate the image. The SonarQube Action is an example of this case. The Action itself wrappers calling a CLI that is provided as a Docker image. As a result, that image (and the CLI) are the actual code being executed. This may require image scanning, further code reviews, or both to ensure that the image (and thus the Action) is safe to use.
There is a further dependency that may not be as obvious. The README file documents that the Action relies on downloading a version of Node.js. We expect that behavior. It provides details about the logic and the source(s) for the downloaded versions. Based on this, I need to review those sources and ensure that I trust the vendor and the approach to providing the binary. In this case, GitHub maintains a set of releases. If the desired version is not found, they fall back to a version provided by NodeJS. If you don’t trust the vendor or the approach, then you may need to use another Action (or fork the code and provide your own binaries).
Review the findings
Take a moment to understand what you’ve discovered and identify any opportunities for further improving the process. This includes adding automation wherever it makes sense. The final decision is often based on understanding the relative risks to the organization and whether there is any mediation that can be done to reduce the risk to an acceptable level. In some cases, that may mean not using the dependency. In other cases, it may mean forking the code and making changes to meet the organization’s needs. In all cases, it means making an informed decision.
The quality of the process should never be judged by the amount of paperwork or bureaucracy involved. Instead, it should be based on the quality of the decisions and the outcomes. It’s a balance between being effective and being efficient. The decision should be based on the organization’s risk tolerance and the relative risks of the dependency. The real key to a successful process is to ensure that the decision is informed and that the risks are understood.
In some of the larger organizations I’ve worked with, they record the findings centrally to ensure that they know what dependencies (and versions) are approved, how they were reviewed, and any necessary considerations for their use. This can be as simple as a spreadsheet or as complex as a full database-based solution. For some organizations, it’s just an issue or markdown document that captures the findings. In this case, if I wanted to document the Action I could use a pull request flow that merges the document into a repository to capture my findings. Once reviewed and approved, the document could be merged into the main branch, ensuring that the information is available to all and providing traceability for who reviewed the dependency and who reviewed the analysis and approved the usage. Workflow automation could also be used to add the Action to the organization’s allow list.
More agile organizations sometimes use a less formal practice with the dependencies, relying on the developer review and ongoing security checks against the related code. Once the code and vendor are initially reviewed, team’s rely heavily on the vendor’s practices to ensure that the code remains functional and secure. They often use automation to continuously verify the code (such as Dependabot) and work with vendors to help integrate in practices that benefit the larger community. If the dependency is a proprietary tool or library, they may rely heavily on documentation from the vendor, the results of any assessments, or additional security tooling and scans.
As with all things in software and architecture, it’s all about finding the right balance. Once you establish your practices, then you can find (and continuously refine) the balance that works for your particular situation.