Kubernetes becomes unavailable after 1 year and requires a full wipe

Open mj3c opened this issue 2 years ago • 1 comments

Description

When Kubernetes is enabled within Docker Desktop, certificates are generated and stored inside ~/Library/Containers/com.docker.docker/pki (or ~/Library/Group Containers/group.com.docker/pki) during the cluster setup. These certificates are valid for 1 year, after which Kubernetes becomes unavailable.

Two workarounds I've found are doing a "reset" of the cluster through the Docker Desktop dashboard, or deleting the pki directory mentioned above and restarting Docker Desktop. However, doing any of that will reset the entire cluster, meaning all PVCs/PVs also get deleted alongside any other resources. If you have local development environments with databases, for example MySQL running as a StatefulSet, this means that:

If 1 year expires, you will not be able to start your environment and backup your data
If you delete the pki directory and restart Docker Desktop, the PV created for the StatefulSet will get deleted and you will lose your data

I also tried generating new certificates and putting them inside the pki directory, but even a single file changing will trigger a reset of the entire cluster.

Why are certificates not renewed automatically before expiry (for example using the kubeadm utility which is available inside the VM), without reseting the cluster? Is there any approach to start Kubernetes after the certs expire without losing data?

Related issue: https://github.com/docker/for-mac/issues/3649

Reproduce

Enable Kubernetes
Create a Persistent Volume in any way (or any other resource)
Delete one of the certs, for example rm ~/Library/Containers/com.docker.docker/pki/apiserver.crt (imagine it expired and has to be replaced)
Quit Docker Desktop and start it again

Expected behavior

Kubernetes should be available and the resources it had should still be present.

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.24.0 (122432)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/mj3c/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/mj3c/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/mj3c/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/mj3c/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/mj3c/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/mj3c/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/mj3c/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/mj3c/.docker/cli-plugins/docker-scout

Server:
 Containers: 39
  Running: 38
  Paused: 0
  Stopped: 1
 Images: 33
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 9
 Total Memory: 11.68GiB
 Name: docker-desktop
 ID: 00603228-0b51-4361-b8c0-316a0df8ef46
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

1D157FD7-632F-466E-A7D3-02FD3F1A98FA/20231010121502

Additional Info

No response

Oct 10 '23 12:10 mj3c