Rootless GitLab Runners

TL;DR: Rootless Docker has become easy! You can also easily run the GitLab Runner binary with rootless Docker without impacting workloads with rootless-kit. Even run buildkit, dind and docker build! See the HOW-TO!

GitLab CI is the biggest CI/CD system after GitHub Actions and is the preferred solution for self-hosted and enterprise SCM-Systems. While GitHub gives you a fresh, full-blown VM for each job, GitLab has the concept of different Executors. While the GitLab-Runner binary manages the communication and job setup, the executor does the actual work.

While there are Shell and SSH executors, container-based executors are often preferred for flexibility, security and reproducibility. While Docker can improve security, it is not know for it in its default configuration, since everything runs as root.

Attack Surface Docker

By default, Docker is started as root and everyone in the docker user group can communicate with the root-owned socket in /var/run/docker.sock. Since the user-ids inside a container map 1:1 to the user-ids on the host, a simple docker run --it --privileged -v /:/host debian can overtake the whole system. Even if the user executing it is not root.

ℹ️ Docker and other containers are NOT virtualization! All processes run on the same kernel as the host and are only isolated by process namespace and cgroup capabilities.

This alone is kind of bad. It gets even worse if you run arbitrary, potentially malicious code from unknown systems. A complete host takeover is only one services: [docker:dind] or a stupid runner config away.

But how can you run jobs rootless and what about docker build jobs that need dind?

Securing Docker & CI

This can be mitigated by not running Docker as root at all. Rootless Docker used to be a pain but has become surprisingly easy on modern systems.

NOTE: The instructions are based on a Debian system but can be adapted to others.

First install Docker according to the official docs:

 1# Install pre-requirements and some extra packages
 2sudo apt update
 3sudo apt install curl ca-certificates curl uidmap apparmor lsb-release \
 4    slirp4netns dbus-user-session fuse-overlayfs systemd-container fuse-overlayfs cifs-utils
 5
 6# Add the docker apt repo
 7sudo install -m 0755 -d /etc/apt/keyrings
 8sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
 9sudo chmod a+r /etc/apt/keyrings/docker.asc
10echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
11    https://download.docker.com/linux/debian \
12    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
13    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
14
15# Finally install docker
16sudo apt update
17sudo apt install docker-ce docker-ce-cli containerd.io \
18    docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras

Now Docker is running as root. Which we don’t want. Disable it via systemd and optionally allow normal users to open ports below 1024.

1# Disable the root docker service
2sudo systemctl restart apparmor.service
3sudo systemctl disable --now docker.service docker.socket
4sudo rm /var/run/docker.sock
5# Allow privileged ports for normal users
6sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0

Now we need the gitlab-runner binary that communicates with GitLab and our executor. By default, the official install scripts also runs gitlab-runner as root. It should also run as a normal user:

1# Create gitlab-runner user and install the GitLab Runner
2sudo useradd --create-home --shell /usr/bin/bash --user-group gitlab-runner
3curl -JOL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh 
4sudo bash script.deb.sh
5sudo apt install gitlab-runner
6sudo systemctl stop gitlab-runner
7sudo systemctl disable --now gitlab-runner

We want to run docker rootless, but the users inside of containers are often root or require root. Since we want all our workloads to work with the new setup we use the user-namespace capability of the Linux kernel. This maps the root user form inside a container to an unprivileged user to the host. So even if an attacker manages to break out of a container, he will never have privileges higher than the newly created gitlab-runner user.

1# Set user-namespace mapping
2sudo echo $(id -u gitlab-runner):100000:65536 >> /etc/subuid
3sudo echo $(id -u gitlab-runner):100000:65536 >> /etc/subgid
4
5# If you run everything headless without a login shell you need to enable lingering
6sudo loginctl enable-linger

We can now switch to the new gitlab-runner user and set up docker and the required configs:

 1# gitlab-runner.service
 2[Unit]
 3Description="GitLab Runner"
 4ConditionFileIsExecutable="/usr/bin/gitlab-runner"
 5After="network.target"
 6
 7[Service]
 8Restart="always"
 9RestartSec=120
10StartLimitInterval=5
11StartLimitBurst=10
12Environment="DOCKER_HOST=unix:///run/user/<<YOUR_GITLAB_RUNNER_USERID>>/docker.sock"
13ExecStart=/usr/bin/gitlab-runner run \
14--config /home/gitlab-runner/gitlab-runner-config.toml \
15--working-directory /home/gitlab-runner \
16--service gitlab-runner --user gitlab-runner
17
18[Install]
19WantedBy="default.target"

 1# gitlab-runner-config.toml
 2concurrent = 4
 3check_interval = 0
 4shutdown_timeout = 0
 5[session_server]
 6  session_timeout = 1800
 7[[runners]]
 8  name = "Rootless Docker"
 9  url = "https://gitlab.com"
10  id = 1234
11  token = "<MY_TOKEN>"
12  executor = "docker"
13  [runners.docker]
14    image = "debian"
15    allowed_pull_policies = ["always", "if-not-present"]
16    allowed_privileged_images = ["moby/buildkit:*", "docker:dind" ]
17    disable_entrypoint_overwrite = false
18    privileged = true
19    oom_kill_disable = false
20    disable_cache = false
21    volumes = ["/cache", "/certs/client"]
22    shm_size = 0
23    network_mtu = 0

We now can start rootless docker and configure gitlab-runner to use it:

 1sudo -i && su gitlab-runner
 2mkdir -p .config/systemd/user /.config/docker
 3
 4# Run rootless docker
 5dockerd-rootless-setuptool.sh install
 6
 7# You can now run docker commands
 8export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock
 9docker run --rm hello-world
10
11# Setup GitLab Runner conf and systemd service with the conf above 
12# Remember to adjust your user-id and set your token
13vim .config/systemd/user/gitlab-runner.service
14vim gitlab-runner-config.toml
15
16# Activate the runner
17systemctl --user daemon-reload
18systemctl --user enable gitlab-runner.service
19systemctl --user start gitlab-runner.service

With the correct token and GitLab server configured you should now see the runner in the GitLab UI.
You now can run any CI-Job with this runner - even dind works!

1rootless-job:
2  tags: [my-rootless-runner]
3  services: [docker:dind]
4  stage: test
5  scripts:
6    - echo "I am running as $(whoami)"

What about Kubernetes

What if you use the Kubernetes GitLab-Executor and run buildkit in Kubernetes?
When you run modern Kubernetes with containerd version >=2.x1 and Kubernetes >= 1.33.x you can use userNamespace with your pods. It is just one parameter. Just set spec.hostUsers to false in your Pods any thats it.

And now?

This is one measurement to avoid security issues that may overtake your CI runners, but it is by far not the only attack that often used. Supply chain security is a wide and deep topic and the upcoming post will explain how to protect your environment from credential threats and other attacks.