Variable Groups in GitHub

One of the great things about working with brilliant people – the collaboration sparks interesting discussions. While exploring some ideas with Josh Johanning recently, we discussed features that we had found helpful in the past for implementing automation. One of those was having variables and variable groups available for build, release, and other automated processes.

Azure DevOps supports variable groups, which makes developing build and release pipelines easier. Having variable groups let you define and manage a set of related variables from a single location. Those values can also be imported into build a release pipelines with minimal effort. Out of the box, GitHub doesn’t have an area to define variables specific to a workflow. The closest technique we have is to define secrets.

Just because we don’t have a UI doesn’t mean we can’t have a feature we like. It just means we have to approach the problem differently. 😄

Why not secrets?

Can’t we just use GitHub repository secrets? In many cases, those work well and can provide the same benefits. The secrets context object in collections of values from the repository or organization. Unfortunately, sometimes we need to be able to output those values. As an example of that, consider a variable with the name of a server or its URL. Secrets are automatically masked to prevent data exfiltration. If those values are masked on output, we can’t easily create reports or display links that include those values.

While it’s very close to the desired functionality, secrets are not always able to provide everything we need.

Composite Actions

There are two approaches we can use for managing groups of values. The first is to use a Composite Action. This is often the easiest approach for organizations with GitHub Enterprise Cloud. Composite Actions are composed of steps in a process, rather than being written in JavaScript or provided as a Docker container.

To expose variables to a job from an Action, the easiest approach is to output the values to a special path, $GITHUB_ENV. By appending additional lines to this file, future steps in the same job can read those values as environment variables.

As an example of how this works, let’s make two environment variables available to other steps in a job:

First, create a repository a repository that contains a single file, action.yml. In that file, define a composite action that appends name/value pairs to $GITHUB_ENV. The contents for this example will look like this:

 1name: 'Global variables'
 2description: 'Configures global variables'
 3
 4runs:
 5  using: "composite"
 6  steps:
 7    
 8    # Create the environment variable SERVER_NAME
 9    - run: echo "SERVER_NAME=prod.my-app.com" >> $GITHUB_ENV
10      shell: bash
11
12    # Create the environment variable PORT_NUMBER
13    - run: echo "PORT_NUMBER=8024" >> $GITHUB_ENV
14      shell: bash

As with any codebase, the latest version is always available by referencing the default branch, main. The repository can also be versioned with tags by creating releases (or by pushing tags directly). This allows us to implement versioning.

Next, this Action must be made available to other repositories. To do this, open the repository settings, select Code and automation > Actions > General, and scroll to the bottom of the page. Update the Access to make the repository accessible from other repositories in the organization:

Other repositories can now include these environment variables by simply running this custom, composite Action. The Action name will be in the format {owner}/{repository}@{ref}. As an example, I can use the latest version of the values from an Action in the repository https://github.com/kenmuse-co/variables by using the following step:

1steps:
2  - uses: kenmuse-co/variables@main

All of the steps after this one will have access to the new variables from expressions (${{env.SERVER_NAME}} or ${{ env.PORT_NUMBER}}) or as standard environment variables (such as $SERVER_NAME in Bash). If the values are changed in the Action, those changes are then available to all of the workflows that use the Action.

If we wanted to manage multiple configurations this way, we could place each configuration in a separate folder. Each folder could be configured as a separate Action with its own YML file.

Files

Because the $GITHUB_ENV file relies on data being appended to it, we can also use a file stored remotely as the source of the name/value pairs. For example, we can download a file from an Azure Blob Storage endpoint, a Gist, or another repository. The file itself would just need to contain name/value pairs on separate lines, similar to this:

1SERVER_NAME=prod.my-app.com
2PORT_NUMBER=8024

To use this file, we can use curl to download the file, redirecting the output to $GITHUB_ENV. The remote file is appended to the file, adding its values as new environment variables. Of course, the file should be stored securely to prevent injecting malicious data! For example, using shared access tokens in Azure (and storing that URL in a Secret).

We can invoke curl by using a run step in our workflow:

1run: curl -sSL https://address.to.file.txt >> $GITHUB_ENV

If this file is stored in a secured, centralized GitHub repository, then a direct download from a URL is not possible. Instead, we can use the GitHub CLI to retrieve the file. To do this, we can call the GitHub repository contents REST API with a personal access token that has read permissions on the repository:

1- run: |
2    gh api -X GET -H "Accept: application/vnd.github.v3+json" \
3       /repos/{OWNER}/{REPO}/contents/{PATH-TO-FILE}` \
4       -jq '.content | @base64d' >> $GITHUB_ENV    
5  env:
6    GH_TOKEN: ${{ secrets.MY_TOKEN }}

The API returns JSON that contains the Base64-encoded file in a property called content. GitHub CLI provides built-in support for jq, enabling us to parse the response. The statement .content | @base64d reads the content property, decodes it, and returns the results. The results are then appended to $GITHUB_ENV using redirection(>>).

Other paths

This is just two of the ways that you can manage variables centrally and pull them into your pipelines. We’ve explored how to manage settings using a remote file, a file in a repository, and managed as an Action. In truth, there are many other approaches possible. You can even combine multiple approaches (such as an Action that reads a file to populate the variables). Actions or steps could also be used to read configuration settings from remote management or configuration systems (such as Azure App Configuration or AWS AppConfig).

Happy DevOp’ing!