IPv4 default route not getting added with static, sbr plugin

[nknkgithub]

Using multus CNI, with sriov plugin. For IP and route assignment static IPAM and sbr plugin is used. But default route is not getting added in routing table 100

Below NAD and pod yaml is applied in k8s cluster

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-test
  annotations:
    k8s.v1.cni.cncf.io/resourceName: "intel.com/pci_sriov_net_physnet0"
spec:
  config: '{
      "cniVersion": "0.3.1",
      "name": "sriov-test",
      "plugins": [
        {
          "type": "sriov",
          "vlan": 961,
          "ipam": {
            "type": "static",
             "routes": [
              { "dst": "0.0.0.0/0", "gw": "10.210.106.254"}
            ]
     }
    },
    {
        "name": "sriov-test-sbr",
        "type": "sbr"
    }
    ]
   }'

Pod yaml

apiVersion: v1
kind: Pod
metadata:
  name: sriov-pod-1
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
                { "name": "sriov-test" ,
                  "ips": ["10.210.106.204/24"],
                  "interface": "test"}

        ]'
spec:
  containers:
  - name: sriov-example
    image: "<image>"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /etc/podnetinfo
      name: podnetinfo
      readOnly: false
    resources:
      requests:
        memory: 1Gi
        intel.com/pci_sriov_net_physnet0: '1'
      limits:
        memory: 1Gi
        intel.com/pci_sriov_net_physnet0: '1'
    command: ["sleep", "infinity"]

  volumes:
  - name: podnetinfo
    downwardAPI:
      items:
        - path: "annotations"
          fieldRef:
            fieldPath: metadata.annotations

default route is not added in routing table 100

sh-4.2# ip rule list
0:      from all lookup local
32765:  from 10.210.106.204 lookup 100
32766:  from all lookup main
32767:  from all lookup default

sh-4.2# ip r s
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link

sh-4.2# ip r s t 100
10.210.106.0/24 dev test proto kernel scope link src 10.210.106.204

Below is the expected route in table 100 default via 10.210.106.254 dev test

sh-4.2# ip rule list
0:      from all lookup local
32765:  from 10.210.106.204 lookup 100
32766:  from all lookup main
32767:  from all lookup default

sh-4.2# ip r s
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link

sh-4.2# ip r s t 100
default via 10.210.106.254 dev test
10.210.106.0/24 dev test proto kernel scope link src 10.210.106.204

[nknkgithub]

Help required!!

[mlguerrero12]

without the sbr plugin, do you see the route?

[nknkgithub]

Hi @mlguerrero12 , without sbr plugin also default route is not getting added in the routing table with requested gateway (ipv4 address)

NAD file without sbr

kind: NetworkAttachmentDefinition
metadata:
  name: sriov-test
  annotations:
    k8s.v1.cni.cncf.io/resourceName: "intel.com/pci_sriov_net_physnet0"
spec:
  config: '{
      "cniVersion": "0.3.1",
      "name": "sriov-test",
      "plugins": [
        {
          "type": "sriov",
          "vlan": 961,
          "ipam": {
            "type": "static",
             "routes": [
              { "dst": "0.0.0.0/0", "gw": "10.210.106.254"}
            ]
     }
    }
    ]
   }'

default route with 10.210.106.254 not getting added in routing table

sh-4.2# ip rule list
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default
sh-4.2#
sh-4.2# ip r s
default via 169.254.1.1 dev eth0
10.210.106.0/24 dev test proto kernel scope link src 10.210.106.204
169.254.1.1 dev eth0 scope link

But when sbr is used and ipv6 route is requested, route is getting added in routing table 100. NAD file with ipv6 route and sbr

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-test
  annotations:
    k8s.v1.cni.cncf.io/resourceName: "intel.com/pci_sriov_net_physnet0"
spec:
  config: '{
      "cniVersion": "0.3.1",
      "name": "sriov-test",
      "plugins": [
        {
          "type": "sriov",
          "vlan": 961,
          "ipam": {
            "type": "static",
             "routes": [
                 { "dst": "0::0/0", "gw": "2005:11:54:4c::7fff"}
            ]
     }
    },
   {
        "name": "sriov-test-sbr",
        "type": "sbr"
    }

    ]
   }'

Pod yaml

apiVersion: v1
kind: Pod
metadata:
  name: sriov-pod-1
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
                { "name": "sriov-test" ,
                  "ips": ["2005:11:54:4c::1/64"],
                  "interface": "test"}

        ]'
spec:
  containers:
  - name: sriov-example
    image: "<image>"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /etc/podnetinfo
      name: podnetinfo
      readOnly: false
    resources:
      requests:
        memory: 1Gi
        intel.com/pci_sriov_net_physnet0: '1'
      limits:
        memory: 1Gi
        intel.com/pci_sriov_net_physnet0: '1'
    command: ["sleep", "infinity"]

  volumes:
  - name: podnetinfo
    downwardAPI:
      items:
        - path: "annotations"
          fieldRef:
            fieldPath: metadata.annotations

default route getting added in table 100

sh-4.2# ip -6 rule list
0:      from all lookup local
32765:  from 2005:11:54:4c::1 lookup 100
32766:  from all lookup main

sh-4.2# ip -6 r s t 100
2005:11:54:4c::/64 dev test proto kernel metric 256 pref medium
default via 2005:11:54:4c::7fff dev test metric 1024 pref medium

We want to get similar default route in table 100 in case of ipv4 address. Expected output with ipv4

sh-4.2# ip r s t 100
default via 10.210.106.254 dev test
10.210.106.0/24 dev test proto kernel scope link src 10.210.106.204

[mlguerrero12]

@nknkgithub, please raise an issue in the sriov cni repo.