[bitnami/nginx-ingress-controller] SSL Termination issue with Nginx ingress controller

[ppaslan]

Name and Version

bitnami/nginx-ingress-controller 9.3.8

What steps will reproduce the bug?

I'm encountering this issue on Azure Aks, Kubernetes 1.23

1. Azure AKS
2. Kubernetes 1.23
3. Installed bitnami nginx-ingress-controller helm chart

Are you using any custom parameters or values?

tcp:
  2222: "privx/ssh-mitm:2222"
  3389: "privx/rdp-mitm:3389"
  1080: "privx/ssh-mitm:1080"
  8443: "privx/nginx:8443"

addHeaders:
  Cache-Control: "private, no-cache, must-revalidate"
  Expires: "0"
  Pragma: "no-cache"
  X-Frame-Options: "SAMEORIGIN"
  X-Content-Type-Options: "nosniff"
  X-XSS-Protection: "1; mode=block"
  Content-Security-Policy: "connect-src 'self' wss://*:*; img-src 'self' data:; default-src 'self'"
  Strict-Transport-Security: "max-age=31536000; includeSubdomains;"

config:
  use-gzip: "true"
  gzip-types: "text/plain text/xml text/css text/comma-separated-values text/javascript application/x-javascript application/atom+xml"
  gzip-min-length: "500"

What is the expected behavior?

I expected to get routed to the correct pod or at least not get the error message in the controller.

What do you see instead?

I can see from the controllers logs that the tls secret is read and added to the local store.

I0913 20:48:02.404739       1 store.go:585] "Secret was added and it is used in ingress annotations. Parsing" secret="privx/privx-tls"
I0913 20:48:02.405355       1 backend_ssl.go:67] "Adding secret to local store" name="privx/privx-tls"

But upon browsing to the LoadBalancer provisioned by AKS i get the following error. I've been trying to google its meaning but has been unable to pinpoint the exact issue.

2022/09/13 20:48:24 [error] 31#31: *14467 [lua] certificate.lua:263: call(): failed to set DER private key: d2i_PrivateKey_bio() failed, context: ssl_certificate_by_lua*, client: 10.244.0.1, server: 0.0.0.0:443
2022/09/13 20:48:24 [error] 34#34: *14469 [lua] certificate.lua:263: call(): failed to set DER private key: d2i_PrivateKey_bio() failed, context: ssl_certificate_by_lua*, client: 10.244.0.1, server: 0.0.0.0:443

Additional information

Describing the ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: privx
    meta.helm.sh/release-namespace: privx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/proxy-ssl-name: nginx
    nginx.ingress.kubernetes.io/proxy-ssl-protocols: TLSv1.2
    nginx.ingress.kubernetes.io/proxy-ssl-secret: privx/proxy-ca-secret
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/ssl-ciphers: AESGCM+EECDH:AESGCM+EDH:AES+EECDH:AES+EDH:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:-DSS:-EDH
    nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
  creationTimestamp: "2022-09-13T16:14:46Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: privx-nginx-https
  name: privx-nginx-https
  namespace: privx
  resourceVersion: "35780"
  uid: 59d3d60c-ebf1-4ced-835e-832907d777b6
spec:
  ingressClassName: nginx
  rules:
  - host: censored.com
    http:
      paths:
      - backend:
          service:
            name: nginx
            port:
              number: 8444
        path: /
        pathType: Prefix
      - backend:
          service:
            name: nginx
            port:
              number: 8444
        path: /auth/login
        pathType: Prefix
      - backend:
          service:
            name: nginx
            port:
              number: 8444
        path: /auth/static
        pathType: Prefix
      - backend:
          service:
            name: nginx
            port:
              number: 8444
        path: /auth/password
        pathType: Prefix
      - backend:
          service:
            name: nginx
            port:
              number: 8444
        path: /auth/change-password
        pathType: Prefix
      - backend:
          service:
            name: nginx
            port:
              number: 8444
        path: /auth/mfa
        pathType: Prefix
  tls:
  - hosts:
    - censored.com
    secretName: privx-tls
status:
  loadBalancer:
    ingress:
    - ip: 10.224.0.4

[fmulero]

Hi @ppaslan

Could you check if your secret privx-tls is a tls-secret? Test for example if you are able to read the certificate using the following command:

kubectl get secret privx-tls -o jsonpath="{.data['tls\.crt']}" | base64 -d | openssl x509 -text -inform pem -noout

Please also check if the key belongs to the certificate

[ppaslan]

Hi @ppaslan

Could you check if your secret privx-tls is a tls-secret? Test for example if you are able to read the certificate using the following command:

kubectl get secret privx-tls -o jsonpath="{.data['tls\.crt']}" | base64 -d | openssl x509 -text -inform pem -noout

Please also check if the key belongs to the certificate

Hi @fmulero

The secret named privx-tls is indeed a tls-secret and the command you supplied works for decoding the certificate. I'm positive the key belongs to the certificate.

[thread-koder]

I have the same problem when I enable proxying can anyone help?

[rafariossaa]

Hi, I am sorry, this same is happening in #12028. We are currently working to fix it. Meanwhile, please use --set image.tag=1.3.0-debian-11-r9 to use an older image without this issue.

[ppaslan]

Hi, I am sorry, this same is happening in #12028. We are currently working to fix it. Meanwhile, please use --set image.tag=1.3.0-debian-11-r9 to use an older image without this issue.

That does indeed fix this temporarily, thank you.

[rafariossaa]

I am moving this to on-hold, to wait #12028 to be fixed.

[justinmchase]

I just got bit by this also, I'm seeing this error:

2022/09/27 15:36:54 [error] 1419#1419: *280958 [lua] certificate.lua:263: call(): failed to set DER private key: d2i_PrivateKey_bio() failed, context: ssl_certificate_by_lua*, client: 198.203.175.175, server: 0.0.0.0:443

With no changes to our certs at all, all we did was upgrade this chart from version 9.2.28 -> 9.3.12 and now every request is failing with that error with the exact same certs as before.

[marcosbc]

Hi folks, we have just released NGINX Ingress Controller chart 9.3.17, bundling bitnami/nginx-ingress-controller:1.4.0-debian-11-r2, which contains a fix for this issue.

In order to be able to keep track of this issue, we are closing it in favor of #12028. If it worked, or if you find any issues, please add a comment there. Thanks!