devspace render showing apiVersion: extensions/v1beta1 for ingress-nginx and can't override it

[cheslijones]

What happened? There doesn't seem to be a way to override the apiVersion: extensions/v1beta1 DevSpace uses for the ingress-nginx controller which could be responsible for the ingress controller not working because in Kubernetes v+1.22 it should use apiVersion: networking.k8s.io/v1.

I've tried overriding this with something like:

annotations:
  helm.sh/chart: component-chart-4.0.16

But it still produces apiVersion: extensions/v1beta1.

What did you expect to happen instead? I was hoping by specifying the component-chart-4.0.16 it would force the use of apiVersion: networking.k8s.io/v1, but devspace render still shows the former.

How can we reproduce the bug? (as minimally and precisely as possible) Install the latest, or any version, of ingress-nginx:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml

Then run devspace render and it will show it is using apiVersion: extensions/v1beta1.

So to get ingress-nginx working it seems like you have to downgrade Kubernetes to < v1.22 and ingress-nginx to < v1.

My devspace.yaml:

version: v1beta11

images:
  # dev and production images
  admin:
    image: ${APP_NAME}/${ADMIN_DEV}
    dockerfile: ${ADMIN}/docker/Dockerfile
    context: ${ADMIN}/
    build:
      buildKit:
        options:
          target: development

deployments:
  - name: ${ADMIN}
    helm:
      componentChart: true
      values:
        containers:
          - image: ${APP_NAME}/${ADMIN_DEV}
        service:
          ports:
            - port: 4001
        ingress:
          rules:
            - host: localhost
              path: /v2/${ADMIN}
              pathType: Prefix
          annotations:
            helm.sh/chart: component-chart-4.0.16

Local Environment:

• DevSpace Version: devspace version 5.18.2
• Operating System: mac
• ARCH of the OS: ARM64 Kubernetes Cluster:
• Cloud Provider: local
• Kubernetes Version: Client: 1.23.3 and Server 1.22.5

Anything else we need to know? Per the ingress-nginx documentation:

Ingress resources evolved over time. They started with apiVersion: extensions/v1beta1, then moved to apiVersion: networking.k8s.io/v1beta1 and more recently to apiVersion: networking.k8s.io/v1.

Here is how these Ingress versions are supported in Kubernetes: - before Kubernetes 1.19, only v1beta1 Ingress resources are supported - from Kubernetes 1.19 to 1.21, both v1beta1 and v1 Ingress resources are supported - in Kubernetes 1.22 and above, only v1 Ingress resources are supported

And here is how these Ingress versions are supported in NGINX Ingress Controller: - before version 1.0, only v1beta1 Ingress resources are supported - in version 1.0 and above, only v1 Ingress resources are

As a result, if you're running Kubernetes 1.19 or later, you should be able to use the latest version of the NGINX Ingress Controller; but if you're using an old version of Kubernetes (1.18 or earlier) you will have to use version 0.X of the NGINX Ingress Controller (e.g. version 0.49).

The Helm chart of the NGINX Ingress Controller switched to version 1 in version 4 of the chart. In other words, if you're running Kubernetes 1.19 or earlier, you should use version 3.X of the chart (this can be done by adding --version='<4' to the helm install command).

No matter what version I install of ingress-nginx, devspace render will always show apiVersion: extensions/v1beta1 is being used. Thus, if I'm using Docker Desktop with Kubernetes version +1.22, then the ingress controller quits working.

# Source: component-chart/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "admin"
  labels:
    "app.kubernetes.io/name": "admin"
    "app.kubernetes.io/managed-by": "Helm"
  annotations:
    "helm.sh/chart": "component-chart-0.8.4"
    "helm.sh/chart": "component-chart-4.0.16"
spec:
  rules:
  - host: "localhost"
    http:
      paths:
      - backend:
          serviceName: admin
          servicePort: 4001
        path: "/v2/admin"
        pathType: "Prefix"

/kind bug

[LukasGentele]

Thanks for opening this issue. The relevant component-chart template is here (see selected lines): https://github.com/loft-sh/component-chart/blob/master/templates/ingress.yaml#L21-L27

Not sure why this is not rendered correctly but we do support the newer ingress api group. Does this just appear for devspace render? Does it deploy the correct api group when using devspace dev/deploy?

[FabianKramm]

Mhh seems like capabilities is not correctly working if helm template is used, seems like we need to change the component chart for this

[LukasGentele]

@FabianKramm We don't need to change the helm chart. Instead, we need to set the --api-versions flag for helm template for Capabilities to work correctly: https://github.com/helm/helm/pull/5392

[cheslijones]

@LukasGentele Hi,

What I'm able to find is the following. Let me know if I can provide any other information:

$ kubectl describe ing admin -n app-dev
Name:             admin
Labels:           app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=admin
Namespace:        app-dev
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)

Rules:
  Host        Path  Backends
  ----        ----  --------
  localhost   
              /v2/admin   admin:4001 (10.1.0.13:4001)
Annotations:  acme.cert-manager.io/http01-ingress-class: nginx
              helm.sh/chart: component-chart-0.8.4
              meta.helm.sh/release-name: admin
              meta.helm.sh/release-namespace: app-dev
Events:
  Type    Reason             Age   From          Message
  ----    ------             ----  ----          -------

And another:

$ kubectl describe pod admin-5867cc4467-5bdtd
Name:         admin-5867cc4467-5bdtd
Namespace:   app-dev
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Thu, 03 Feb 2022 07:52:15 -0800
Labels:       app.kubernetes.io/component=admin
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=devspace-app
              pod-template-hash=5867cc4467
Annotations:  helm.sh/chart: component-chart-0.8.4
Status:       Running
IP:           10.1.0.13
IPs:
  IP:           10.1.0.13
Controlled By:  ReplicaSet/admin-5867cc4467
Containers:
  container-0:
    Container ID:   docker://59b000c769073046fca63e9cc039527013efba767acc491c724b759dddbf0864
    Image:          app/admin-dev:kTOqPiN
    Image ID:       docker://sha256:15eaed105dafaf3be2b1e034ae1ff6530ffc5c9d5394c8af3f8a82cbe35397dc
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 03 Feb 2022 07:52:16 -0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvc6v (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-tvc6v:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  25m   default-scheduler  Successfully assigned app-dev/admin-5867cc4467-5bdtd to docker-desktop
  Normal  Pulled     25m   kubelet            Container image "app/admin-dev:kTOqPiN" already present on machine
  Normal  Created    25m   kubelet            Created container container-0
  Normal  Started    25m   kubelet            Started container container-0

Also, I see this message in the ingress-nginx-controller:

I0203 15:52:23.204929       8 store.go:420] "Ignoring ingress because of error while validating ingress class" ingress="app-dev/api" error="ingress does not contain a valid IngressClass"

[LukasGentele]

@cheslijones Actually, what you found is a separate issue. However not one that DevSpace can fix. You need to an an ingress class to the ingress in devspace.yaml for the values of this deployment:

values:
  ingress:
    ingressClassName: xyz

To figure out xyz, run: kubectl get ingressclass

To prevent having to set ingressClassName for every ingress, you could also mark one of the ingress classes in your cluster as default, by running kubectl edit ingressclass xyz and adding this annotation:

ingressclass.kubernetes.io/is-default-class: "true"

[cheslijones]

@LukasGentele Thanks for pointing that out that it is an unrelated issue.

I haven't had to specify that before, but it looks like now that I have added it that error has cleared up.

Is there anything I can send you related to the apiVersion? The only place I'm able to find it clearly stated is with devspace render.

devspace dev and checking various svc, pod, and ing about the only thing I can find is helm.sh/chart: component-chart-0.8.4. I think this should be helm.sh/chart: component-chart-4.x.x though.

[gradox2020]

I have the same problem. Do you have any solution or workaround for this Thanks very much

[FabianKramm]

@gradox2020 this should be fixed with the newer component chart version