Using GitLab to manage Kubernetes access

Recently I have been tasked to share access to a Kubernetes test cluster with other teams. Easy: Add them to a group in your OIDC provider and set appropriate RBAC rules. Task done - no need to write this.

BUT:

Unfortunately I can’t have an OIDC provider for non-production clusters… because. Sighs…

Okay what other options are there for Kubernetes authentication:

• Webhook Token Authentication
• Authenticating Proxy
• Static token file
• X509 client certificates

Both webhook authentication and an authentication proxy add considerable overhead and ultimately also need to be plunged into a user registry that I can not use or have control of.

So what about static token files?
In practice, I’ve never seen them outside of test setups. It requires placing a file with the username, secret and groups on every controlplane node. The effort required to manage and sync such a file, containing static credentials in plain text, is just nuts. Even more absurd, the api-server has to be restarted every time this file is changed! And since these tokens have no built-in expiration date, you have to delete an entry and restart the api-server if a credential needs to be invalidated. So this is a hard pass!

So X509 client certificates it is?
The X509 client certificates approach does not require the api-server to be restarted and provides a built-in expiration mechanism. Client certificates are a common approach to establishing a secure communication between two entities, it is often also referred to as mutual TLS (mTLS), where both the client and the server have to identify themselves via a certificate. One-directional TLS (where just the server is ensuring its authenticity) is part of almost every website we visit.
TLS is considered very secure since, it uses asymmetric encryption with private keys, with a complexity between 1024 and 4098 bits, depending on the algorithm. Kubernetes uses mTLS for almost all of its components to establish trusted and secure communication between every part of the cluster. Kubernetes clusters created with kubeadm also create a admin.conf which also uses X509 client certificates to authenticate against the Kubernetes api-server.
These X509 client certificates are the default way to authenticate against a Kubernetes cluster. For easier usage, Kubernetes base64 encodes the certificate and the private key and puts them inside the well-known YAML kubeconfig. The file is most likely located at ~/.kube/config.

I have gone the X509 client certificates way and if you are interested, below is a rough draft on how you can automate user config creation. But there is a catch too Sighs…x2

 1# Bits for the key
 2KEY_SIZE=4096
 3# Username
 4USERNAME=MY_USER
 5# Groups
 6GROUP_1=TEAM_A
 7GROUP_2=TEAM_B
 8
 9# Creating key and certificate signing request
10openssl req -new -newkey rsa:$KEY_SIZE -nodes -keyout k8s-client-1.key -batch -out k8s-client-1.csr -subj "/CN=${USERNAME}/O=${GROUP_1}/O=${GROUP_1}/emailAddress=${USERNAME}@example.com"

 1apiVersion: v1
 2kind: Config
 3current-context: docker-desktop
 4clusters:
 5    - cluster:
 6          certificate-authority-data: LS0...S0K
 7          server: https://Kubernetes.docker.internal:6443
 8      name: docker-desktop
 9contexts:
10    - context:
11          cluster: docker-desktop
12          user: docker-desktop
13      name: docker-desktop
14users:
15    - name: docker-desktop
16      user:
17          client-certificate-data: LS0...g==
18          client-key-data: LS0...o=

The Problems with static credentials

While mTLS is technical very secure, it still suffers from problems organizational standpoint. From a modern standpoint, certificates are still long-lived credentials. Thanks to the exemplary work done by the lets encrypt project, most certificates are issued with shorter lifetimes compared to the previously common validity of 1+ years. Still, a default of three months is a heck longer than the one-hour default lifetime of an OIDC ID-Token.
It is also possible to revoke certificates, but just like the creation process, it is not that easy to implement this process. Even when these security aspects are not considered, client certificates become hard to manage in larger or dynamic teams. You must have a way to revoke access from a retired team member. When a user needs to be added, the whole certification creation process has to be done again. Authorization should be done on group level basis, but you can’t add a group to a certificate after it was issued. This is not flexible and manageable in any larger organization. For this reason, dynamic authentication processes - like OIDC - are often preferred.

I have a great article about OAuth2 and OIDC coming up soon on my companies website - stay tuned

Alternatives

So what are my alternatives?
The said cluster in deployed in a very restrictive network environment. Ingress is only allowed from specific clients within a corporate network and only specific ports and traffic. Egress is only possible via a proxy. But we needed to deploy from our GitLab (which is on a different network) instance to our cluster.
Other people solved this by creating GitLab runners within the Kubernetes cluster and saving the kube-client-credentials-conf as a CI secret. DONT DO THIS!

You are pulling in random container images from the internet to your cluster and run arbitrary scripts from CI in your critical application environment. You are risking your cluster health, the confidently of secrets and are giving up a considerable amount of compute. And you also have to maintain the runners. This also violates the ISO 27001 norm since now access is not personalized anymore and auditing is really hard.

GitLab Agent

Quite some time ago I stumbled upon the GitLab Agent. It’s a small application that creates a reverse tunnel between your control-plane and GitLab instance via websockets. This allows you to use any runner in any network location to securely communicate over the GitLab Instance websocket tunnel, via the GitLab agent to your Kubernetes cluster. GitLab automatically creates short-lived tokens (CI_JOB_TOKEN) and only allows authorized users to use the tunnel in CI.

You can install the agent easily with:

 1helm repo add gitlab https://charts.gitlab.io
 2helm repo update
 3helm upgrade --install test gitlab/gitlab-agent \
 4    --namespace gitlab-agent \
 5    --create-namespace \
 6    --set image.tag=v17.0.0 \
 7    --set config.token=glagent-xxx \
 8    --set config.kasAddress=wss://gitlab.example.com//-/Kubernetes-agent/

Now you can create a file in you git repo with the path .gitlab/agents/<agent-name>/config.yaml and even share the cluster access with other Git projects:

1ci_access:
2  projects:
3    - id: my-team/serviceX
4    - id: external-team1/projectA
5    - id: external-team2/projectB
6  groups:
7    - id: qa/loadtests

No secrets needed, secure and controlled access to the cluster.

WAIT…That wasn’t the problem I wanted to solve. I need to share access with people - not with CI!

Good News: This also works for people.

You can just add the following to your .gitlab/agents/<agent-name>/config.yaml:

 1user_access:
 2  access_as:
 3    user: {}
 4  projects:
 5    - id: global/groups/admins
 6  groups:
 7    - id: my-team
 8    - id: qa/loadtests

Now GitLab allows all users that are in these projects or groups to access the cluster endpoint via the agents tunnel.
The important part in that config is the access_as.user property. The Agent will use impersonation to act like the GitLab user that uses the tunnel. That means you can easily set fine-grained permissions via RBAC to only allow the qa group read access while the admins have cluster-admin roles. If you are a masochist, you can also apply the RBAC rules per user.

The Solution

Now, all users can configure their kubeconfigs to use the cluster endpoint https://gitlab.exmaple.com/-/Kubernetes-agent and use their GitLab credentials to authenticate.
Basic permissions are handled by GitLab and the fine-grained Kubernetes permissions are controlled via RBAC.

If a team member retires, just remove them from the GitLab group and the access is gone. A new team member needs onboarding, add them to GitLab and he already has access to the cluster. Easy management and even ISO 27001 compliant 🙌

 1apiVersion: v1
 2kind: Config
 3current-context: gitlab-cluster-via-agent-42
 4clusters:
 5    - cluster:
 6        server: https://gitlab.exmaple.com/-/Kubernetes-agent
 7      name: gitlab
 8contexts:
 9    - context:
10        cluster: gitlab
11        user: gitlab-agent-42
12      name: gitlab-cluster-via-agent-42
13users:
14    # Users can use a pat to access the cluster
15    - name: gitlab-agent-42
16      user:
17        token: "pat:42:XXX"
18    # Or use the glab cli for just in time credentials with a 24h expiry
19    - name: gitlab-agent-42-glab
20      user:
21        exec:
22            apiVersion: client.authentication.k8s.io/v1
23            args:
24                - cluster
25                - agent
26                - get-token
27                - --agent
28                - "42"
29            command: glab
30            env:
31                - name: GITLAB_HOST
32                  value: https://gitlab.example.com
33                - name: GITLAB_CLIENT_ID
34                  value: XXX
35            installHint: "To access the cluster install glab. Instructions at https://gitlab.com/gitlab-org/cli#installation.\n"
36            interactiveMode: Never
37            provideClusterInfo: false
38

Side Notes

This is not the only solution to access private clusters. But this did the job and security is fine with it. Jump Hosts, VPNs and SSH tunnels are also commonly used but they are often a nightmare to manage or to set up. A notable alternative is tailscale which also creates a tunnel to you tailnet and allows for user authentication. Unfortunately, tailscale is not used at my current gig.

While these are solutions, I still would recommend to just hook your OIDC provider such as ActiveDirectory, LDAB or Keycloak into the Kubernetes api-server. It is also a much more standardized solution and should be the primary source for all user contexts.
To my surprise not many managed Kubernetes offerings support setting the OIDC args for the api-server. So this might after all be a good solution if your Kubernetes provider is lacking.