Kubernetes is continuously innovating and adding new functionality. For example, in December 2023, Kubernetes 1.29 introduced the beta version of native sidecar containers ( KEP #753). As a beta feature, that means it is available by default. It’s also getting closer to being promoted to GA (which will happen next year with v1.33).
In this post, we’ll look at the problems sidecars are designed to solve and how to use them.
Why we need native sidecars
A sidecar container is simply a container that runs alongside the main application container in a pod, usually to provide some sort of supporting functionality (such as logging or service proxies). Historically, sidecars have been a pattern rather than a feature. Unfortunately, sidecars have had limited guarantees. The sidecar could finish before the main application, suddenly revoking a needed service. It could also start after the application (since application containers start in parallel), leaving the application without the service it needed. One of the bigger pain points is that a sidecar could keep the pod alive, even after the main application has completed.
There’s one other challenge. Sidecars are often smaller than the application. This can make them a perfect candidate for the out-of-memory killer when node pressure forces Kubernetes to terminate containers. Killing the sidecar will suddenly deprive the main application of the service, leading to unpredictable or undesirable behaviors.
Clearly, there were reasons for the community to want a better solution.
How they work
At their core, sidecar containers are just a special case of init containers. An init container runs before the main container starts, and is traditionally used for setup tasks that need to complete before the main containers can run. Init containers are executed sequentially.
Native sidecar containers have an characteristic: they continue to run until after the main application containers have completed. This happens because of a single change to the deployment specification: they have a restartPolicy of Always. These containers automatically restart if terminated, support probes, and don’t block the main application containers from starting or stopping.
Native sidecars have another benefit. The OOM score for the sidecars is adjusted to match or exceed the application containers it serves. This reduces the likelihood that it is killed due to memory pressure. This also helps to ensure that the sidecar is available for the entire life of the application containers, even if the node is under memory pressure.
As an init container, the sidecar will start before the application containers in the order it is defined. While normal init containers complete and terminate before the next one can start, sidecars indicate that they are complete using a startup probe or, if that’s not present, a postStart handler. If neither are present, the container is considered to be started if it is running.
When the application containers are done, the sidecars will terminate. The sidecar’s preStop handler is invoked when the pod starts to terminate. Next, the sidecars terminate sequentially (via a SIGTERM signal), starting with the latest one to start. If the graceful termination period ends before all of the containers have terminated, the remaining sidecars will receive a SIGTERM. Two seconds later, all of the containers will be terminated by a SIGKILL. In general, this behavior means that exit codes from sidecars should be ignored.
Probing the container
Unlike other init containers, native sidecars have the ability to use the Kubernetes probes. Essentially, probes are a way for Kubernetes to understand more about the container and it’s health. There are three types of probes.
The first type is the liveness probe. It continuously runes to test whether the container needs to be restarted. If this probe fails, the container is killed and restarted. A common example is detecting a deadlock or “hung” application. If the application cases tp be responsive, it can be restarted to resolve the issue. This one requires some care; if the container is less responsive under high load, it might accidentally trigger a restart, compounding the problem.
The second type is the readiness probe. It determines when the container is ready to start accepting traffic. This probe continuously checks whether the container is able to handle requests. This can allow the container to indicate when it cannot handle requests due to re-establishing connections, performing cache cleanup, or other tasks. When it fails, the container can no longer receive traffic.
The final type is the startup probe. This determines when the container has completed the startup process. This probe is only used during the startup process. This probe is often an important part of configuring a sidecar. When a regular init container starts, it runs to completion. After that, the next init container runs. Native sidecars are long-running and automatically restart any time the stop, so they require a different behavior.
Once the sidecar container has started, Kubernetes executes any registered postStart handlers. After that, the next init container can begin executing. By default, Kubernetes considers a sidecar to be started as soon as it begins. However, when a startup probe is configured, Kubernetes will use that to determine when the sidecar has successfully started. When the probe succeeds, the container has started. If the probe fails, Kubernetes will continue to retry the probe until it succeeds or reaches a configured failureThreshold (at which point the container is killed and restarted).
Probes can be implemented a few different ways:
- httpGet
- Makes an HTTP call to a specific path and port, optionally providing specific headers. Codes in the range 200 - 399 indicate a success, while any other code indicates a failure.
- tcpSocket
- Opens a TCP connection to a specific port. If the connection is successful, the probe is considered to be successful. Any failure to connect is considered a failure.
- grpc
- Determines success or failure by using the gRPC health checking protocol.
- exec
- Runs a command inside the container. A exit status code of zero indicates success, while any other code indicates a failure. Because this can create multiple processes, it can increase the CPU usage on a node more than the other approaches. At the same time, it allows for logic that can be executed using scripts or command lines without relying on a long-running service.
Each of these options provides a specific way for Kubernetes to query the state of the container, and each one can require slightly different settings. These options may it easy for a sidecar to implement health and startup checks, giving it feature parity with traditional application containers.
Next steps
Native sidecars are a powerful feature that can help you build more robust applications. They can provide services or features to the main application, along with stronger guarantees about its lifecycle, health, and OOM characteristics. This makes them a great fit for situations where the application needs to rely on a supporting service.
In my next post, we’ll look at a practical example of how to modernize a Kubernetes deployment to use a native sidecar.