Ken Muse

Universal Packages on GitHub With ORAS

Category:Tags:Published: July 14, 2023 Reading Time: 6 min

Most package management systems support some kind of “universal artifacts” storage. Azure Artifacts supports “universal packages”, Artifactory supports “generic repositories”, and Nexus has “raw repositories”. But what about GitHub? From all appearances, it would seem this is an oversight. Although not explicitly documented, it turns out it’s fully supported.

Adopting open standards

The first thing to understand is that for many years, languages, platforms, and projects created their own package management system. With no standardized way to store and manage packages, solutions proliferated. In 2018, ORAS (OCI Registry as Storage) was born with the idea of using the OCI Distribution APIs to unify package management for Cloud Native Computing Foundation (CNCF) projects. It joined the CNCF in 2021.

ORAS provides a standardized way to create, store, and search for arbitrary packages or files. By being built on top of standardized container registries, it makes those systems automatically compatible … and that means it’s supported by GitHub.

You may not have realized it, but many tools already use ORAS. For example:

  • Helm Charts
  • Homebrew
  • Azure Bicep Modules
  • Docker image signatures and SBOMs
  • Dev Container Features
  • Singularity
  • Notation
  • Policy

And if you’re using GitHub Actions, the ORAS CLI is part of the Linux runner images.

Logging in

For GitHub, you just need to provide a PAT for the password. Any user name can be used.

1oras login ghcr.io -u github

Or, integrating with the GitHub CLI:

1gh auth refresh -s write:packages,repo,delete:packages
2gh auth token | oras login ghcr.io --password-stdin -u github

Getting started (fast!)

The most basic use of ORAS simply relies on using push to upload the artifact and pull to download it. Artifacts are tagged/versioned like Docker images, making it easy to discover a specific artifact package and its versions.

To create an artifact with a README.md file and all of the files in the binaries folder:

1oras push ghcr.io/kenmuse/awesome-tool:1.0 --artifact-type: application/vnd.kenmuse.mytool.config.v1 ./README.md ./binaries

This creates an artifact in my personal GitHub Packages namespace. It has the name awesome-tool and version (tag) 1.0. The provided artifact-type is similar to a file extension and uniquely identifies my format. It makes it easy to identify artifacts of the same type.

To retrieve the files in the artifact and unpack them on the file system:

1oras pull ghcr.io/kenmuse/awesome-tool:1.0

By default, files are stored with the media type application/vnd.oci.image.layer.v1.tar. Folders are stored compressed as application/vnd.oci.image.layer.v1.tar+gzip. As you’ll see below, there’s a way to override the media type for files and folders.

Discovery

An important part of package management is being able to query the packages and their versions. For example, I can list the available repositories using oras repo ls ghcr.io/kenmuse (which would return awesome-tool). I can also list all of the versions (tags) for that artifact using oras repo tags ghcr.io/kenmuse/awesome-tool.

Getting Attached

ORAS also supports “attaching” artifacts to each other, creating a relationship. This is commonly used to support signatures and SBOMs (by attaching those to the related image). To create an attachment, simply use oras attach to specify the file(s) to attach and the target of the attachment. For example, to attach a PDF handbook to my artifact, I could use:

1oras attach --artifact-type application/vnd.kenmuse.handbook.config.v1 --image-spec v1.1-image ghcr.io/kenmuse/awesome-tool:1.0 ./handbook.pdf

To see the complete tree of attachments with a given item and the relationships, you can use: oras discover ghcr.io/kenmuse/awesome-tool:1.0 -o tree. To find a specific artifact type, use oras discover --artifact-type application/vnd.kenmuse.handbook.config.v1 ghcr.io/kenmuse/awesome-tool:1.0.

To view the attached artifact’s manifest, I can use its SHA:

1oras manifest fetch ghcr.io/kenmuse/awesome-tool@sha256:613ab688e8fa67ac5f8f1917042f290699856b5f1f4ad243fdd7c46208dc58bd

This returns the details:

 1{
 2  "schemaVersion": 2,
 3  "mediaType": "application/vnd.oci.image.manifest.v1+json",
 4  "config": {
 5    "mediaType": "application/vnd.kenmuse.handbook.config.v1", // Notice that without the v1.config
 6    "digest": "sha256:7e8649e94fb4fc21fe77e8310c060f61caaff8a9629b6df6b688e8fa67ac5f8f",
 7    "size": 2
 8  },
 9  "layers": [
10    {
11      "mediaType": "application/vnd.oci.image.layer.v1.tar", 
12      "digest": "sha256:9629b6df6b688e8fa67ac5f8f1917042f290699856b5f1f4ad243fdd7c46208",
13      "size": 16229,
14      "annotations": {
15        "org.opencontainers.image.title": "handbook.pdf"
16      }
17    }
18  ],
19  "subject": {    // This is the artifact to which it is attached
20    "mediaType": "application/vnd.oci.image.manifest.v1+json",
21    "digest": "sha256:6b688e8fa67ac5f8f1917042f290699856e774e745b5c6c3a6a35b6476579568",
22    "size": 1105
23  },
24  "annotations": {
25    "org.opencontainers.image.created": "2023-07-15T04:22:05Z"
26  }
27}

Within that manifest, I can also find the SHA for the uploaded PDF. I can use that information to download the a specific file blob:

1oras blob fetch -o -  ghcr.io/kenmuse/awesome-tool@sha256:9629b6df6b688e8fa67ac5f8f1917042f290699856b5f1f4ad243fdd7c46208

This restores the file automatically. I can choose to set a different media type for the file to make it easier to discover. To do that, just add :{TYPE} to the file name:

1oras attach --artifact-type application/vnd.kenmuse.handbook.config.v1 --image-spec v1.1-image ghcr.io/kenmuse/awesome-tool:1.0 ./handbook.pdf:application/pdf

Using the ORAS CLI

It’s easily to get started with the CLI. First, ORAS requires you to define a root artifact type name. Generally, the format for a type is application/vnd.{COMPANY}.{TYPE}.{OPTIONAL_SUBTYPE}.config.v{VERSION}+{FORMAT}. The purpose of this type is similar to a file extension – it provides a way for tools to understand how to interact with the type. The opencontainers/artifacts repo provides a more detailed explanation. The guide from Steve Lasker is also worth a read. They dive deep into the reasoning and recommendations.

This name is a MIME Media Type, with vnd indicating it’s a vendor-defined type. The vendor (company) is then specified. After that, a short name for the type and any optional subtype. The config component refers to the fact an artifact always starts with an Image Config (config) that can store metadata. Because formats and contents may change over time, a version number is provided to make it easy to understand the version of the contents. If a Config is not provided, the artifact type stops there. Otherwise, it will typically end with +json or +yaml to indicate the format of the configuration data. For maximum compatibility, JSON is recommended.

These are best practices to ensure maximum compatibility and usability, but not required. For example, image/jpeg is also a valid artifact type. That said, consistent use of a type makes discovery and long-term management easier. For example, Helm uses application/vnd.ccnf.helm.chart.config.v1+json. The packaged chart is stored with the type application/vnd.cncf.helm.chart.content.v1.tar+gzip (which indicates the chart content format is v1, and it contains one or more files in a GZIP-compressed TAR file). They document these media types in more depth on the Helm site.

Moving right along

A related feature in ORAS is the ability to use oras copy to promote an artifact tree or copy it to a different registry. Microsoft provides a walkthrough using Azure Container Registry.

More to come

The specification for artifacts and the ORAS tooling continues to evolve. It’s a community open-source project, so it relies on feedback and usage in the community to help it grow and mature. Feel free to get involved and contribute.

Hopefully this has given you some insights into how to use this powerful tool and some of the features available. There’s a lot more functionality to explore, so take some time to read the docs.

Happy DevOp’ing!