Sysbox v0.6.1 daemon-set installer fails in GKE 1.24.10-gke.2300 with DataplaneV2 (Cilium) or Calico

[kevholmes]

It appears as though during the sysbox install process (via your daemonset) I get some extra cri-o configs 100-crio-bridge.conf, 200-loopback.conf co-located in /etc/cni/net.d that breaks my GKE networking and introduces a subnet for the node that's not correct according to what's been configured in GKE by the 10-gke-ptp.conflist config file already there.

When pods come back online after the kubelet restart on that node, a number of them including the sysbox-installer get this incorrect CIDR provided in 100-crio-bridge.conf and they go into crash loop backoff state as they cannot reach anything on the network (which is correct, there are no firewall/policy rules allowing traffic from this new CIDR that came out of thin air.)

For reference, here's the content of 100-crio-bridge.conf that gets placed into /etc/cni/net.d and breaks the networking for pods running on the node I'm trying to install sysbox on:

{
    "cniVersion": "0.3.1",
    "name": "crio",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "hairpinMode": true,
    "ipam": {
        "type": "host-local",
        "routes": [
            { "dst": "0.0.0.0/0" },
            { "dst": "1100:200::1/24" }
        ],
        "ranges": [
            [{ "subnet": "10.85.0.0/16" }],
            [{ "subnet": "1100:200::/24" }]
        ]
    }
}

I've tried restarting various services on the node, along with just restarting the node entirely after deleting these extraneous files from /etc/cni/net.d (while ensuring I keep 10-gke-ptp.conflist) but no dice, I just get a ton of messages like this:

Apr 19 21:27:34 gha-std-runners-2-baae0455-8zr5 kubelet[25451]: E0419 21:27:34.993101   25451 pod_workers.go:965] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?" pod="kube-system/sysbox-deploy-k8s-98fpk" podUID=3287c25f-d5f2-477e-a934-918722a0908a

@rodnymolina for viz as you asked me to bring this in as an Issue to track.

[rodnymolina]

Thanks @kevholmes, will take a look.

[jamonation]

Same issue here with same GKE version. Nodes got upgraded and similar symptoms. However, I don't have a file 10-gke-ptp.conflist:

root@gke-k8stty-pool-1-afb28145-8549:/etc/cni/net.d# ls
10-calico.conflist  100-crio-bridge.conf  200-loopback.conf  calico-kubeconfig

I'm going to try using one of the untagged images from before the v0.6.1 release will report back.

[rodnymolina]

Right, this is probably a consequence of sysbox-deploy deamonset being unaware of pre-existing cnis with non-default configurations. Will take a look when have a chance.

[jamonation]

I think I've figured out the issue - it looks like the default routes/subnets in 100-crio-bridge.conf assume IPv6 is enabled and available based on the docs description.

Removing the route and range gets things working again on my nodes (edit file, check with jq because missing commas etc. will blow up crio, then systemctl restart crio:

root@gke-k8stty-pool-1-afb28145-8549:/etc/cni/net.d# git diff 100-crio-bridge.conf
diff --git a/100-crio-bridge.conf b/100-crio-bridge.conf
index d17c67a..ca9a5fd 100644
--- a/100-crio-bridge.conf
+++ b/100-crio-bridge.conf
@@ -9,12 +9,10 @@
     "ipam": {
         "type": "host-local",
         "routes": [
-            { "dst": "0.0.0.0/0" },
-            { "dst": "1100:200::1/24" }
+            { "dst": "0.0.0.0/0" }
         ],
         "ranges": [
-            [{ "subnet": "10.85.0.0/16" }],
-            [{ "subnet": "1100:200::/24" }]
+            [{ "subnet": "10.85.0.0/16" }]
         ]
     }
 }

But I'm still trying to confirm since I've got some wedged GKE ingress/neg bits to sort out still.

[jamonation]

Well ignore what I wrote above, was a red-herring. Turns out after removing both 100-crio-bridge.conf, 200-localhost.conf, and adding "/home/kubernetes/bin" to crio.conf I've got things working:

  [crio.network]
    plugin_dirs = ["/opt/cni/bin/", "/home/kubernetes/bin"]

The issue being the calico CNI files live in /home/kubernetes/bin, and that having the upstream CNI bridge configuration included was assigning pods IPs that weren't reachable.

[jamonation]

I can't find the source for /opt/sysbox/scripts/sysbox-deploy-k8s.sh but I think detecting GKE should be relatively easy to add. For example, if a node is part of a cluster, a GET/HEAD to the internal metadata endpoint should return an HTTP 200:

[root@sysbox-deploy-k8s-879zb etc]# curl -si -H "Metadata-Flavor: Google" 169.254.169.254/computeMetadata/v1/instance/attributes/cluster-name
HTTP/1.1 200 OK
Content-Type: application/text
Metadata-Flavor: Google
Server: GKE Metadata Server
Date: Sat, 29 Apr 2023 14:23:00 GMT
Content-Length: 6

<cluster_name>

From there, removing the various /etc/cni/net.d/ files, and patching crio.conf should work. Something like this?

rm /etc/cni/net.d/100-crio-bridge.conf /etc/cni/net.d/200-localhost.conf
dasel put string -f ${host_crio_conf_file} -p toml -m 'crio.runtime.network.[]' "/opt/cni/bin/"
dasel put string -f ${host_crio_conf_file} -p toml -m 'crio.runtime.network.[]' "/home/kubernetes/bin"

[joncbenderkh]

This also happens on GKE v1.25.8-gke.1000 without dataplane v2.

Applying the fix @kevholmes posted and restarting all pods seems to fix the issue.

[jglick]

Installation failed for me in v1.26.5-gke.1400 though the error message is different and it is not clear to me if the problem is the same. Eliding some IAM setup I think is irrelevant:

gcloud container clusters create … \
  --release-channel=stable \
  --image-type=ubuntu_containerd \
  --machine-type=n1-standard-4 \
  --workload-pool=$(gcloud config get-value core/project).svc.id.goog \
  --num-nodes 3 \
  --min-nodes 3 \
  --max-nodes 9 \
  --spot \
  --addons=GcpFilestoreCsiDriver
kubectl label nodes --all sysbox-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml
kubectl -n kube-system rollout status daemonset sysbox-deploy-k8s

(Stable release channel pending #704. Image type according to this and this. Installation adapted from this.)

The daemonset rollout apparently succeeds, but then no other pods start successfully (3 node(s) had untolerated taint {sysbox-runtime: not-running}; all are still sysbox-runtime=installing), and then the sysbox-deploy-* pods and some other pods are stuck in ContainerCreating with

Warning  FailedCreatePodSandBox  29m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_sysbox-deploy-k8s-…_kube-system_…_0(…): error checking pod kube-system_sysbox-deploy-k8s-… for CNI network "200-loopback.conf": cache file path requires network name (""), container ID ("…"), and interface name ("eth0")

Not understanding what I am doing, on one node I tried deleting /etc/cni/net.d/200-loopback.conf (the only other file in that directory was 10-gke-ptp.conflist) and editing /etc/crio/crio.conf to include

  [crio.network]
    plugin_dirs = ["/opt/cni/bin/", "/home/kubernetes/bin"]

but this just leads to a different error, like the original shown in this issue

Warning  FailedCreatePodSandBox  1s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_sysbox-deploy-k8s-…_kube-system_…_0(…): No CNI configuration file in /etc/cni/net.d/. Has your network provider started?

[yachub]

Well ignore what I wrote above, was a red-herring. Turns out after removing both 100-crio-bridge.conf, 200-localhost.conf, and adding "/home/kubernetes/bin" to crio.conf I've got things working:

  [crio.network]
    plugin_dirs = ["/opt/cni/bin/", "/home/kubernetes/bin"]

The issue being the calico CNI files live in /home/kubernetes/bin, and that having the upstream CNI bridge configuration included was assigning pods IPs that weren't reachable.

I can also confirm this issue and that the fix above resolves it. GKE 1.24.14-gke.1400 (Ubuntu with containerd (ubuntu_containerd))