How to investigate problems with Kubernetes containers with “Kubectl Debug”

Kubernetes logo

It can be difficult to diagnose issues when running Kubernetes workloads. You might get lucky and find the cause in your application logs via kubectl logs command. However, in some cases, you can’t avoid a live debugging session where you interactively engage with your Pods to uncover issues.

The kubectl debug command simplifies these debugging tasks by providing a new ephemeral container in your Pod. This can be used to verify the Pod environment so you can start troubleshooting issues that occur in your existing containers.

Preparing to use Kubectl Debug

kubectl debug was launched with v1.18 of Kubernetes and Kubectl. Relies on ephemeral containers are available in your cluster. Ephemeral containers became a beta feature in Kubernetes v1.23 and are now enabled by default. You will need to manually activate port function if your cluster is running an older version of Kubernetes.

Ephemeral containers are intended for transient tasks where you need to temporarily attach an additional container to an existing Pod. This is ideal for debugging operations where you want to accurately inspect a Pod without affecting live container instances.

Most container images have no debugging tools; installing them in a running container would change its environment and potentially cause side effects. Attaching an ephemeral container to your Pod is a safer way to debug that gives you a clean working environment. You can use a heavier image that includes all the necessary tools.

Although ephemeral containers become part of their Pod host, there are still some differences you should be aware of. Ephemeral containers do not support port binding, samples, or resource reservations, as they are only temporary in nature. They will never restart automatically and cannot be changed once created. A full list of supported capabilities is available in the documentation.

Using Kubectl Debug

Before proceeding, create a basic implementation to use for testing purposes:

$ kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created

Then use get pods command to find the Pod name of your deployment:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-55649fd747-qsnr2   1/1     Running   0          5s

The floor of our deployment is called nginx-55649fd747-qsnr2.

Now you can use kubectl debug command to start a debugging session on your Pod:

$ kubectl debug -it --image=ubuntu:20.04 nginx-55649fd747-qsnr2

The command syntax is similar to a hybrid of kubectl create and kubectl debug. The unnamed argument supplied to the command identifies an existing Pod to attach to. The --image argument specifies the image to use for the new container. We use ubuntu:20.04 here to access the familiar commands included with the Ubuntu Linux distribution.

The -it flag is equivalent to --stdin --tty. Including these arguments will allocate a TTY to the container, attach it, and connect the stdin stream to your terminal. This gives you an interactive shell in your new container.

Now you can run your debugging tasks from your ephemeral container.

Copy sub

Another way to use kubectl debug is with a --copy-to argument. This creates a copy of the target Pod and adds the ephemeral container to the copy. The original capsule is left intact.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug nginx-555649fd747-qsnr2

This feature gives you even more assurance that changes made during debugging will not directly affect your production application.

Copying a Pod also allows you to enable the namespace sharing process. This makes existing processes in your Pod visible to your ephemeral container. It also cannot be used with existing containers spec.shareProcessNamespace usually set to false. running kubectl debug with --copy-to and --share-processes flag will enable process sharing in the copied Pod, making this procedure much more intuitive:

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --share-processes nginx-555649fd747-qsnr2

The list of processes visible to your ephemeral Ubuntu container will now include an NGINX process:

$ ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    9 root      0:00 nginx: master process nginx -g daemon off;

This process still runs in the separate NGINX container in your Pod. Namespace sharing also provides access to the target container’s file system via the /proc:

$ ls /proc/9/root/etc/nginx
conf.d fastcgi_params mime.types modules nginx.conf ...

Therefore, copying a Pod like this is a powerful debugging tool. You can easily inspect Pod files and processes using a separate container that is prepared with familiar tools.

Optional arguments

The --copy-to flag always leaves the original Pod intact by default. You can make the operation act as a placeholder instead of using --replace. This will stop the first Pod.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --replace nginx-555649fd747-qsnr2

Kubernetes will schedule the copied Pod to any available node. This can be problematic if you want to ensure a consistent test environment. Add --same-node will schedule the copy to the existing Pod node, eliminating any differences that may exist between the machines in your cluster.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --same-node nginx-555649fd747-qsnr2

Another useful option is --env to set additional environment variables in your ephemeral container. You may need to use this to configure debugging tools or to override values ​​inherited from your target Pod.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --env EDITOR=/usr/bin/nano nginx-555649fd747-qsnr2

Finally, remember that containers created by kubectl debug they don’t have to be interactive. You can easily run one-time commands against your Pods using kubectl exec-similar syntax. The --attach an argument is maintained to control whether your shell is associated with the container when you are not working with -i (--stdin).

$ kubectl debug --image=ubuntu:20.04 --copy-to nginx-debug --share-processes --attach true nginx-555649fd747-qsnr2 -- ls /proc/9/root/etc/nginx
conf.d fastcgi_params mime.types modules nginx.conf ...

Conclusion

Ephemeral containers and kubectl debug provides a simplified debugging experience for Kubernetes workloads. You can run commands inside a Pod using an image different from your regular containers. This allows you to access debugging tools that are not included in your application image.

kubectl debug can also create copies of Pods and share their processes with the original. This mechanism allows you to inspect the processes in the target Pod’s containers from a separate ephemeral container over which you have full control. It provides more advanced debugging options when you need to interrogate running processes.

Related Posts

Leave a Reply

Your email address will not be published.