Testing Kubernetes Operators and Controllers With Minikube

I recently needed to test some changes in GitHub’s Actions Runner Controller (ARC), but I didn’t want to have to deploy an updated image to a registry just to be able to validate the changes. I wanted to be able to quickly and easily test locally. I decided to use Minikube to assist me.

If you’re not familiar with Minikube, it essentially lets you run a small Kubernetes cluster on your machine. In my case, hosted using Docker on a Mac. This allows me to work with Kubernetes without needing to deploy an environment on Azure or AWS. This makes it great for testing and experimentation. In addition it has the added bonus of being able to run from inside a Dev Container, so I can use it on Codespaces. This makes it an especially good choice for testing and experimenting with Kubernetes pods, operators, and controllers. It can also make it easy to validate Helm charts.

This post will explore a few different ways you can use minikube to help with the development and testing of these components. While I’m going to use ARC as a practical example to illustrate the techniques, this post is not intended to be a comprehensive guide into building, customizing, or deploying ARC. If you are interested in understanding those practices, review the Contributing.md file in the ARC repository.

Keeping it simple with images

Most people that work with Kubernetes are familiar with the fact that Kubernetes will cache the images it pulls from a registry to improve performance. Minikube is very similar, but adds an additional feature. You can interact with its image store directly. This allows you to build or load an image into Minikube and then use it in your cluster. This also makes it possible to preload a minikube instance with a set of images and use it offline.

Another benefit of this approach is that I can build an image inside of minikube or build an image locally, then load it into minikube.

Creating an image

I wanted to see if a pull request for ARC would create an image that resolved an issue I was seeing. To do this, I first cloned the branch and repository to my local machine. I then compiled the image using the following command:

1 docker buildx build --build-arg VERSION=0.9.3 -t custom-scale-set-controller:custom .

This builds a new version of the image with the components versioned based on a VERSION argument that compiles them as version 0.9.3. The command also adds a tag to the image, naming it custom-scale-set-controller:custom. For most cases, the tagging simply provides a nice way to identify the image and its version.

Loading the image into Minikube

The minikube image store includes both the images used to bootstrap minikube and any images loaded to run in Kubernetes. Once an image is loaded into the store, minikube will resolve requests for that image from there rather than trying to pull it from a remote registry. You can use minikube image ls to see a list of the images used by an instance of minikube.

To load a cistp, local Docker image into Minikube, you use the image load command. It will copy the image into Minikube’s image store. That image can then be used by the cluster (or to enhance minikube itself).

1minikube image load custom-scale-set-controller:custom

There are multiple other options for loading and building images in minikube, including building inside of minikube, using ssh, and deploying a cluster registry. More on that in a moment.

Using the image

Once the image is loaded, it’s ready to be used in a deployment. You can use a traditional Kubernetes deploy or update and deploy a Helm chart. In this case, I’m using ARC which relies on a Helm chart. This requires updating values.yaml for the ARC controller to set image.repository to my custom image.

1image:
2  repository: "custom-scale-set-controller"
3  tag: "custom"

If you don’t specify a tag, ARC will default to using the version associated with the chart. As a result, that value can usually be omitted.

Testing it out

When this Helm chart is deployed, it will create the related Kubernetes resources. That includes downloading and deploying the images specified in the charts (or deployment YAMLs) unless they already exist in minikube. This process makes it easy to test the changes in an isolated minikube environment.

Loading images into a minikube registry

To build and test operators locally often requires working with Helm. This requires an OCI registry. Thankfully, minikube has an addon that can provide a local registry, the minikube registry add-on. Instead of loading the image manually, both it and the Helm chart can be pushed to the minikube registry. That allows testing the deployment process, the packaging, and the image in a single environment.

The registry add on requires some configuration. First, you have to start minikube with an additional command line parameter (--insecure-registry "10.0.0.0/24). This allows it to use and trust the internal registry without a TLS certificate. This won’t work for an existing minikube configuration, so making this change requires deleting any existing minikube cluster.

If you’re working with a Mac or Windows and running minikube on Docker, then you will need to run a socat proxy. The full details are here. Since I’m running on a Mac, I just need to run this command to start a proxy:

1docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"

The proxy makes localhost:5000 available locally and within the cluster, allowing it to be shared between the two. Next, build the image and push it to the registry. That can be done by tagging it with the localhost:5000 registry and pushing it:

1docker buildx build --build-arg VERSION=0.9.4 -t localhost:5000/custom-scale-set-controller:custom .
2docker push localhost:5000/custom-scale-set-controller:custom

Once this is done, the image can be referenced in minikube as localhost:5000/custom-scale-set-controller:custom.

Loading a Helm chart into the registry

Of course, this example would not be complete without also considering how to package and push a Helm chart. To do that, you perform the normal packaging process (helm package <path-to-chart>) and then publish the resulting tarball to the registry using helm push <path-to-tarball> <registry>.

For example, I want to deploy the gha-runner-scale-set chart. I also want to specify a version and app version (to override the values in the Chart.yaml). Changing to that directory, I could run:

1helm package . --app-version 0.9.4 --version 0.9.4
2helm push ./gha-runner-scale-set-0.9.4.tgz oci://localhost:5000/helm-charts

This will package the chart and upload it to the minikube registry. I could the refer to that chart using the same OCI URL to deploy to the cluster. For this example: oci://localhost:5000/helm-charts/gha-runner-scale-set. Since the registry is shared with the local environment, it’s accessible from both the local machine and the minikube cluster.

Conclusion

As you can see, minikube provides several features that make developing and testing charts and images quite easy. In addition, the rich command line provides a lot of control over the environment and its features. If you’re working with Kubernetes, consider spending time to understand this tool and the many features it can provide.