The recurring problem of the Metrics Server and Insecure Kubelet certificate

What is this about:

Currently the kubelet, the client agent coordinating container lifecycle on each node, also runs a server, using a self signed certificate by default. Therefore any client connecting the the kubelet server, most prominent the metrics server, will not be able to verify the servers identity.

Current state:

Currently if the kubelet starts and no --tls-cert-file and --tls-private-key-file are specified it generated a self-signed pair and uses it for its server component according the kubelet docs
The is a feature feature flag called RotateKubeletServerCertificate which would allow the use of serverTLSBootstrap. This uses the native CertificateSigningRequest (csr) resource wich is part of the certificates.k8s.io/ api. When serverTLSBootstrap is enabled the kubelet creates a csr via the kubernetes certificates api for its server component. Users can either approve the csr manually with kubectl certificate approve <my-csr> or create a rolebinding wich allows for auto-approve. The resulting cert is singed by the clusters ca if the kube-controller has access to the ca cert and key (most likely it has).
The process is described in the tls-bootstrapping documentation, this provides the prerequisites for a trust chain verification.

The problem:

With self-singed certificates there is no way to reliable verify the kubelet’s server identity. Even the metrics-servers docs says that passing --kubelet-insecure-tls is not a solution for production.
The recent addition serverTLSBootstrap is an approvement but the control over the created csr is very limited to non existed. Most Kubernetes setups are not trivial. Nodes have multiple IPs and may be available over one ore move dns names, let alone IPv6 addresses. From my experience and according to other posts the created kubelet csr does not contain any SANs which could identify the server. The CN ist set to system:node:<name> wich is NOT a valid DNS name.
This lack over the certificate details also results in the lack of the possibility to verify the identity of the kubelet server. Binging us back the the need for --kubelet-insecure-tls for all clients talking to the kubelet. Even managed kubernetes offerings have this problem.

The Proposal:

Kubernetes admins and vendors should have an easy possibility to crate trusted kubelet server certificates.
This can be achieved either via the certificate generation phase of kubeadm or via passing extra args to the kubelet tls setup to include additional SANs.

 1# Existing behavior with etcd and kube-api-server
 2apiVersion: kubeadm.k8s.io/v1beta3
 3kind: ClusterConfiguration
 4apiServer:
 5  certSANs:
 6    - "10.100.1.1"
 7    - "ec2-10-100-0-1.compute-1.amazonaws.com"
 8etcd:
 9  local:
10    imageRepository: "registry.k8s.io"
11    serverCertSANs:
12      -  "ec2-10-100-0-1.compute-1.amazonaws.com"
13---
14apiVersion: kubeadm.k8s.io/v1beta3
15kind: InitConfiguration
16nodeRegistration:
17  name: "ec2-10-100-0-1"
18  kubeletExtraArgs:
19    node-ip: "10.100.0.1"
20    # Proposal: ether as an arg
21    serverCertSANs: "10.100.0.1,ec2-10-100-0-1.compute-1.amazonaws.com"
22  # Proposal: or as an own kubeadm nodeRegistration property
23  serverCertSANs:
24    - "10.100.0.1"
25    - "ec2-10-100-0-1.compute-1.amazonaws.com"

Personal I’m in favor of adding an extra arg to the kubelet binary to include a list of additional SANs. This would also allow profit users not using kubeadm to have a properly configured kubelet server certificate.

The Workaround:

For personal testing setups I use ansible to prepare my nodes and bootstrab a cluster. One the certificate generation phase is complete I can generate my onw private key and certificate request and sing it with the clusters ca. Though I don’t recommend this for production, this is why we need a more managed and automated approach.

References

• Kubernetes Docs TLS Bootstrap
• Kubernetes Docs kubeadm troubleshoot tls
• GitHub metrics-server error - missing SANs
• GitHub kubernetes node - missing SANs
• GitHub kubernetes singed serving certificates
• Reddit - kubeadm not including SANs