I recently updated from Debian 10 (Buster) to 11 (Bullseye) and since then my Jenkins setup inside Docker is not working anymore, as Jenkins tries to find out if it is running in a docker container by checking /proc/self/cgroup.
Normally /proc/self/cgroup inside a docker container would look something like this:
12:rdma:/ 11:perf_event:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 10:freezer:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 9:memory:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 8:cpuset:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 7:devices:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 6:net_cls,net_prio:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 5:hugetlb:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 4:pids:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 3:cpu,cpuacct:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 2:blkio:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 1:name=systemd:/docker/a2ffe0e97ac22657a2a023ad628e9df837c38a03b1ebc904d3f6d644eb1a1a81 0::/system.slice/containerd.service
but since I updated to Debian 11 it looks pretty small:
0::/
As Jenkins is not recognizing anymore that it is running inside a docker container itself, it starts other build containers with the wrong arguments.
Question
The simple question would be: Is this a bug?
But the real question might be what am I doing wrong? I cannot find anyone else with this problem, so it might be a misconfiguration or anything similar.
I reinstalled Docker, removed any configuration and I even tried downgrading Docker to 20.10.6 as this is the last version I know was working under Debian 10, but none of that changed anything.
I don’t have a clue on how to approach this problem any further. It already took me a full day to find out that the problem was not Jenkins itself (nearly got crazy reading Jenkins logs). I am hitting bedrock right now, so any help and any input is really appreciated!
Jenkins stuff
For those interested in the Jenkins part, here Jenkins checks if it is running inside a container: https://github.com/jenkinsci/docker-workflow-plugin/blob/b174d46226ef1095903f2e789355a3b216b46dda/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java#L347
Jenkins thinking it is not running inside a container will log something like this:
Jenkins does not seem to be running inside a container $ docker run -t -d -u 0:0 -w /var/jenkins_home/workspace/myrepo_master -v /var/jenkins_home/workspace/myrepo_master:/var/jenkins_home/workspace/myrepo_master:rw,z -v /var/jenkins_home/workspace/myrepo_master@tmp:/var/jenkins_home/workspace/myrepo_master@tmp:rw,z -e ******** ... my-awesome-build-container cat
And thus mounting /var/jenkins_home from the host system, where Jenkins has no access to from inside its container.
While the log output on Debian 10 (and Ubuntu 20.04) looks something like this:
Jenkins seems to be running inside container 7814083762a1bed51dec2f468c6ee07c978a0b6377e347c3ed7dc23393feac11 $ docker run -t -d -u 0:0 -w /var/jenkins_home/workspace/myrepo_master --volumes-from 7814083762a1bed51dec2f468c6ee07c978a0b6377e347c3ed7dc23393feac11 -e ******** ... my-awesome-build-container cat
and starting the build container with the correct volume using --volumes-from.
Edit: The Jenkins plugin is now fixed since version 528.v7c193a_0b_e67c by PR#280: https://github.com/jenkinsci/docker-workflow-plugin/pull/280
Advertisement
Answer
The change in behavior is due to that debian uses cgroups v2 starting with Debian 11/Bullseye. The docker engine itself supports cgroups v2 since v20.10.x.
This means, as soon as you have a distribution that uses cgroups v2 and a recent version of the Docker engine, you cannot get the container id with your method.
I’ve opened a similar question to find an alternative method: How to get docker container ID from within the container with cgroup v2
The only way I know to get the id is by using the docker api, but that is not an elegant solution if you just want to know if the process runs inside a container. (And may pose a security risk if you expose the docker socket inside the container)
For now as a workaround you could manually signal the process that it is run inside a container environment, e.g. by specifying an environment variable on container creation.