Clusters broken with Minikube 1.26.0

[ahamilton55]

Hi. This is mainly a FYI but clusters are broken when using minikube version 1.26.0. Minikube 1.25.2 and earlier versions work as expected. Downgrading to at least minikube 1.25.2 should fix issues.

The issue seems to be with minikube updating containerd in the 1.26.0 release which no longer works with the older style of insecure registry configurations. Minikube needs to be updated to utilize the newer style of registry configuration and ctlptl shouldn't need any changes.

You might see errors where containers stay in the ContainerCreating state and messages like

[event: pod atm/atmr-db-migrate-q75zf] (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b7f3f841cd2efa4a0fd20a2a79e4a44745db4e10d0eccce2804c0a87e5d63932": plugin type="bridge" name="crio" failed (add): failed to set bridge addr: could not add IP address to "cni0": permission denied

[nicks]

thanks for reporting! ya, we'll likely need to redo the registry support for the new containerd

[ahamilton55]

With the change there aren't actually any changes that need to happen on the ctlptl side and it actually simplifies the repo updates that are done after the cluster is created. I'm trying to get the fix for this landed and I've been testing locally with ctlptl to make sure it's relatively easy.

[nicks]

we do edit it a bit ourselves (see: https://github.com/tilt-dev/ctlptl/blob/main/pkg/cluster/admin_minikube.go#L206) but i will keep an eye out for your minikube change!

(https://github.com/kubernetes/minikube/issues/14480)

[ahamilton55]

I put in a PR for the minikube things https://github.com/kubernetes/minikube/pull/14482.

Also, I think if you look at the blame on that line in admin_minikube.go you'll see my name so I have some understanding around the changes that are needed here. This fix should simplify all of that away for the most part because the --insecure-registry flag for minikube will actually setup a working registry for ctlptl. The sed command won't be applied because that section of the containerd configuration was removed so it won't find a string match to make the change against.

[nicks]

now that minikube v1.27 is out, i'm going to take a pass at trying to get the registry working again.

[ahamilton55]

This worked for me this morning. Built locally and then tilt up came up with running pods without any other changes.

The updates on the minikube side remove the need for additional updating by ctlptl on startup and just work out of the box now.

diff --git a/pkg/cluster/admin_minikube.go b/pkg/cluster/admin_minikube.go
index ef850b2..10b142c 100644
--- a/pkg/cluster/admin_minikube.go
+++ b/pkg/cluster/admin_minikube.go
@@ -21,6 +21,7 @@ import (

 // minikube v1.26 completely changes the api for changing the registry
 var v1_26 = semver.MustParse("1.26.0")
+var v1_27 = semver.MustParse("1.27.0")

 // minikubeAdmin uses the minikube CLI to manipulate a minikube cluster,
 // once the underlying machine has been setup.
@@ -86,7 +87,10 @@ func (a *minikubeAdmin) Create(ctx context.Context, desired *api.Cluster, regist
        if err != nil {
                return err
        }
-       isRegistryApi2 := v.GTE(v1_26)
+       isRegistryApi2 := false
+       if v.GTE(v1_26) && v.LT(v1_27) {
+               isRegistryApi2 = true
+       }

        clusterName := desired.Name
        if registry != nil {
@@ -135,12 +139,13 @@ func (a *minikubeAdmin) Create(ctx context.Context, desired *api.Cluster, regist
        }

        // https://github.com/tilt-dev/ctlptl/issues/239
+       // Skip minikube versions 1.26.0 and 1.26.1
        if registry != nil {
                if isRegistryApi2 {
                        return fmt.Errorf(
                                "Error: Local registries are broken in minikube v1.26.\n" +
                                        "See: https://github.com/kubernetes/minikube/issues/14480 .\n" +
-                                       "Please downgrade to minikube v1.25 until it's fixed.")
+                                       "Please upgrade to minikube v1.27.")
                }
                args = append(args, "--insecure-registry", fmt.Sprintf("%s:%d", registry.Name, registry.Status.ContainerPort))
        }

~/go/bin/ctlptl create cluster minikube --registry=ctlptl-registry --kubernetes-version=v1.23.10
😄  minikube v1.27.0 on Darwin 12.6
✨  Using the docker driver based on user configuration
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
📦  Preparing Kubernetes v1.23.10 on containerd 1.6.8 ...
    ▪ kubelet.max-pods=500
    ▪ kubelet.cni-conf-dir=/etc/cni/net.mk
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /usr/local/bin/kubectl is version 1.25.1, which may have incompatibilites with Kubernetes 1.23.10.
    ▪ Want kubectl v1.23.10? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Switched to context "minikube".
 🔌 Connected cluster minikube to registry ctlptl-registry at localhost:50947
 👐 Push images to the cluster like 'docker push localhost:50947/alpine'
cluster.ctlptl.dev/minikube created