It’s no secret that I’m a fan of development containers. They allow me to quickly create consistent, repeatable environments that I can share. With the introduction of Features, we can now quickly and easily share complex functionality and layer that on top of an image. Unfortunately, this can introduce an additional performance cost.
Why so slow?
The first time a Feature is run, a new Docker layer is created by running an install.sh script to configure the container environment. This layer can be cached, but it isn’t quite the same behavior as a traditional Docker layer. With a traditional Docker approach, we might call install.sh as part of a RUN step. The resulting changes would create a layer, and the results could be cached for later reuse. From that moment on, we can use the image (or rebuild the image using cached results).
Features are somewhat different. While they operate on the container, the dev container CLI is ultimately responsible for handling the process. The teams behind these tools have put significant effort into trying to ensure that Feature layers are cached when they are run. This helps to make that a one-time cost (or less, if you are using prebuilds).
Unfortunately, not all aspects of Docker are available to us during these processes. For example, you can’t take advantage of using RUN-scoped caches to assist with package management. While that might be fine in most cases, there are cases where this is less tolerable.
The first path
The easiest path is to re-think the need to build images on the fly. In most cases, the definition of the image itself could be built ahead of time in a parallel repository or environment. From there, it can be easily shared and updated. With this approach, we often have multiple images:
- A base image that is maintained for one or more projects.
- A production image built on the base image and optimized for execution.
- A development image which provides additional SDKs and runtime support. This image is often (but not always) build on top of the production image. These images can include all additional functionality required, including those that might be otherwise distributed as Features.
The advantage of this approach is that developers can both test against a production image and develop in container that is closely tied to production. By separating these concerns, the builds can occur asynchronously compared to development. Updating the local development environment just requires downloading the latest image, so it is incredibly fast.
There are two downsides to this approach. First, someone must now maintain the process for installing the additional functionality. Instead of using community-offered features, you have locally managed scripts. As functionality or best practices change, these scripts must be updated. The other downside is that you lose a separation of concerns. Features can contribute additional mounts and lifecycle hooks to the underlying container. When this is managed directly as part of the container, the combined set of hooks and mounts must be managed by each devcontainer.json that uses the image.
The second path
The first approach makes a lot of sense for core images, but doesn’t always work for additional tooling. Sometimes the advantages of distributing a Feature outweigh the build costs. In this case, we need to find a way to improve the performance. One way we can improve performance is by managing our own resource cache as part of the Feature.
A case-study: Jupyter Notebooks
As an example, let’s consider a simple feature designed to create a dev container that runs on Alpine and supports dev containers. We’ll walk through one set of approaches, but there are always others available. As always, feel free to message me and share your favorites and suggestions!
If you’re new to Features, start by reading the Dev Container Features series.
Let’s start with a simple devcontainer-feature.json:
1{
2 "id": "jupyter-alpine",
3 "version": "0.0.1",
4 "name": "jupyter-alpine",
5 "customizations": {
6 "vscode": {
7 "extensions": [
8 "ms-python.python",
9 "ms-toolsai.jupyter"
10 ]
11 }
12 }
13}This Feature will include the
Python and
Jupyter extensions for Visual Studio Code. For these to work (and to have access to a few helpful libraries), we need to create an install.sh script to ensure the libraries are available. This script gets run automatically to configure the Feature.
1#!/bin/sh
2sudo apk --no-cache --update-cache add gcc gfortran \
3 python3 python3-dev py3-pip \
4 build-base wget freetype-dev libpng-dev \
5 openblas-dev linux-headers libffi-dev cairo-dev
6sudo su -s /bin/sh ${_CONTAINER_USER} << EOF
7pip install -U pip setuptools wheel
8pip install -U --user ipykernel notebook matplotlib numpy scipy \
9 svglib seaborn jupyter nbconvert pandas
10EOFThis particular script runs the pip install process as the defined containerUser (which is provided from the _CONTAINER_USER environment variable). Alternatively, if I wanted to install the Python packages globally, I could instead use:
1#!/bin/sh
2sudo apk --no-cache --update-cache add gcc gfortran \
3 python3 python3-dev py3-pip \
4 build-base wget freetype-dev libpng-dev \
5 openblas-dev linux-headers libffi-dev cairo-dev
6sudo -H pip install -U pip setuptools wheel
7sudo -H pip install -U --user ipykernel notebook matplotlib numpy scipy \
8 svglib seaborn jupyter nbconvert pandasFor most teams, this might be satisfactory. What if, however, one or more of those dependencies requires substantial time to create the binary artifacts? With Docker, we might consider using a RUN and a type=cache mount to optimize the process. For example:
1RUN --mount type=cache,target=/root/.cache/pip sudo -H pip install ...If we had a static image, we would instead invoke Docker and provide a volume mount that points to the global (or user cache).
Feature mounts
What about creating a mount in the feature definition, such as this?
1 "mounts": [
2 {
3 "type":"volume",
4 "source":"jupyter-alpine-pipcache",
5 "target":"/root/.cache/pip"
6 }
7 ],Unfortunately, mounts are not available to us during the initial build. They are added to the dev container and only become available later in the lifecycle. As a result, the install.sh can’t take advantage of the mount.
Later …
The key to understanding our other option is ’later’, during the lifecycle events. The mounts are not initially provided during the build process, but they can be access during the container finalization lifecycle events. That means that we can use the mounts as long as we execute the script during one of these phases:
onCreateCommand- executed on first start, but no access to user assets or secrets.updateContentCommand- executed whenever new content is available and limited similar toonCreateCommand.postCreateCommand- executed once the container has been assigned to a user for the first time, providing it access to user secrets, assets, and permissions.postStartCommand- executed each time the container starts.postAttachCommand- executed each time a tool attaches to the container.
Each of these phases has access to the container’s mounts. As a result, the feature can cache or use content that is preserved between restarts. Because Docker will automatically recreate missing volumes, the initial folder will be empty during the first run. Subsequent runs, however, will see the content from previous executions, enabling them to use the previously built wheels. Of course, this can be more broadly applied to other scenarios and languages.
Volumes are also available outside of the lifecycle events. If we register an entrypoint for the Feature, that will be executed after the build as part of the container runs. That means the entrypoint can also have access to a volume to provide it state across (re)builds or environments.
VS Code uses a similar approach to cache and install the VS Code server and any requested extensions. By reusing a volume called vscode, it can minimize the time required to download requested dependencies (which can ultimately use several GiB of disk space). It can then hook into lifecycle events to configure the IDE settings and extensions. This process takes advantage of downloads and updates from other dev containers, so multiple environments can benefit from this process.
A day in the life
Understanding the dev container lifecycle events makes advanced customizations and functionality possible for Features. It enables you to create and share state, improving overall application performance. Combined with the native Docker layer caching, powerful Feature functionality can be created while minimizing user wait times.
Happy DevOp’ing!