[BUG] TLS certificate is not being updated for appgw container listener when kubernetes secret holding certificate has been updated

[AdamMachera]

Describe the bug When using ALB controller in version mcr.microsoft.com/application-lb/images/alb-controller:1.0.0 and testing how certificate renewal would work it seems that there is a problem.

I have imported new version of pfx containing SSL into keyvault. I have two pods that are referencing this secret and they are being reloaded by https://github.com/stakater/Reloader Updated version of the secret sslcert (type kubernetes.io/tls) is present and is updated by SecretProviderClass.

To verify it I'm exporting the content of the secret kubectl -n mynamespace get secret sslcert -o json -o=jsonpath="{.data.tls.crt}" | base64 -d > fullchain.cer kubectl -n mynamespace get secret sslcert -o json -o=jsonpath="{.data.tls.key}" | base64 -d > somekey.key and I see that fullchain.cer has the newest certificate.

However the alb-controller pod is not updating gateway listener, it is still using old one. I had to execute kubectl delete pod alb-controller-5b5f549cd-ghtwn -n azure-alb-system

Once new instance of the alb-controller pod is created it starts using latest certificate.

To Reproduce use helm mcr.microsoft.com/application-lb/images/alb-controller:1.0.0

Create gateway

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: auth-gateway
  namespace: somenamespace
  annotations:
    alb.networking.azure.io/alb-id: $RESOURCE_ID
spec:
  gatewayClassName: azure-alb-external
  listeners:
  - name: https-listener
    port: 443
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: Same
    tls:
      mode: Terminate
      certificateRefs:
      - kind : Secret
        group: ""
        name: sslcert
  addresses:
  - type: alb.networking.azure.io/alb-frontend
    value: $FRONTEND_NAME
EOF

Add Http route to your backend. Backend should have reloader setup to update the sslcert if it changes in the keyvault. Check in the browser what certificate version is used. It should be using old one. Import into keyvault latest version of SSL certificate. alb-controller is not reloading ssl certificate for gateway listener so browser is still using old version of the SSL certificate. Once we delete alb-controller pod (leader) it will pickup new SSL certiticate.

[yarashagarwal]

I can confirm the same issue.

[t-vynamani]

[Important] [Feedback] AKS alb-controller monitoring metrics Describe your scenario Understand customer tracking of alb-controller and what actionable metrics needs to prioritized during troubleshooting and outages in an improved dashboard

Feedback The Application Gateway for Containers team is eager to enhance your platform for tracking Azure Kubernetes cluster metrics. The focus of this survey is to understand customers' current methods for gathering, viewing, and examining AKS cluster metrics. Link: https://forms.office.com/r/jsx0U0BUy2

Five minutes should be expected to complete this form. It would be greatly appreciated if you could respond by June 14th, 2024. Thank you!

[JackStromberg]

Hey folks (@AdamMachera, @yarashagarwal)!

Sorry about the delays on this!

This issue was addressed in v1.0.2 of the controller back in May. Can you please confirm the update solved the issue?

Cheers! Jack

[JackStromberg]

Closing this as we have not heard back and did release a change to directly address this issue. Please file another issue if applicable. Cheers!