datawire
Helm chart fails when referencing licenseKey secret - pods stuck in ContainerCreating
I'm running into an issue with the latest edge-stack helm chart. The pods get stuck in ContainerCreating due to a missing secret. This seems to happen when configuring the "licenseKey" attributes in values passed to the chart in order to reference a secret we're using for the license key we purchased.
Steps to reproduce:
- Create namespace and a secret with license key purchased
- Install helm chart for edge-stack - pass in values to reference secret name and key where license key is stored:
licenseKey:
annotations: {}
createSecret: false
secretName: edge-stack-license
value: license-key
Results:
- Chart deploys successfully
- Pods are stuck in
ContainerCreatingstate
%
kubectl get pods -n edge-stack
NAME READY STATUS RESTARTS AGE
edge-stack-58444544bb-bnfzd 0/1 ContainerCreating 0 10m
edge-stack-58444544bb-hp87z 0/1 ContainerCreating 0 10m
edge-stack-58444544bb-jnq74 0/1 ContainerCreating 0 10m
edge-stack-58444544bb-ltlqt 0/1 ContainerCreating 0 10m
edge-stack-58444544bb-pd9cp 0/1 ContainerCreating 0 10m
edge-stack-58444544bb-sktf4 0/1 ContainerCreating 0 10m
edge-stack-agent-5d5879b568-lk4k8 1/1 Running 0 10m
edge-stack-redis-56cdf99d69-zxhxk 1/1 Running 0 10m
- Events that show up on the pods:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m42s default-scheduler Successfully assigned edge-stack/edge-stack-58444544bb-bnfzd to ip-10-64-80-135.us-west-2.compute.internal
Warning FailedMount 7m41s kubelet MountVolume.SetUp failed for volume "edge-stack-secrets" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 5m39s kubelet Unable to attach or mount volumes: unmounted volumes=[edge-stack-secrets], unattached volumes=[kube-api-access-2dn86 ambassador-pod-info edge-stack-secrets]: timed out waiting for the condition
Warning FailedMount 89s (x10 over 7m40s) kubelet MountVolume.SetUp failed for volume "edge-stack-secrets" : secret "edge-stack" not found
Warning FailedMount 69s (x2 over 3m24s) kubelet Unable to attach or mount volumes: unmounted volumes=[edge-stack-secrets], unattached volumes=[ambassador-pod-info edge-stack-secrets kube-api-access-2dn86]: timed out waiting for the condition
- Secrets in namespace (the secret ("edge-stack") that the pods are trying to use for a volumeMount is missing):
%
kubectl get secrets -n edge-stack
NAME TYPE DATA AGE
default-token-fpjzt kubernetes.io/service-account-token 3 23m
edge-stack-agent-token-284tp kubernetes.io/service-account-token 3 10m
edge-stack-license Opaque 1 23m
edge-stack-token-fswfj kubernetes.io/service-account-token 3 10m
sh.helm.release.v1.edge-stack-license.v1 helm.sh/release.v1 1 23m
sh.helm.release.v1.edge-stack.v1 helm.sh/release.v1 1 10m
Expected results:
- Chart deploys successfully
- Pods show up in a running state and secret should be present:
%
kubectl get pods -n edge-stack
NAME READY STATUS RESTARTS AGE
edge-stack-58444544bb-29fch 1/1 Running 0 13m
edge-stack-58444544bb-5zh6f 1/1 Running 0 13m
edge-stack-58444544bb-qclf9 1/1 Running 0 13m
edge-stack-58444544bb-r98w6 1/1 Running 0 13m
edge-stack-58444544bb-tqgpz 1/1 Running 0 13m
edge-stack-58444544bb-wz6hf 1/1 Running 0 13m
edge-stack-agent-5d5879b568-xvnsf 1/1 Running 0 13m
edge-stack-redis-56cdf99d69-wl9h5 1/1 Running 0 13m
%
kubectl get secrets -n edge-stack
NAME TYPE DATA AGE
ambassador-internal Opaque 2 14m
default-token-fpjzt kubernetes.io/service-account-token 3 65m
edge-stack Opaque 1 14m
edge-stack-agent-token-9q8lf kubernetes.io/service-account-token 3 14m
edge-stack-token-xbsrc kubernetes.io/service-account-token 3 14m
fallback-self-signed-cert kubernetes.io/tls 2 14m
sh.helm.release.v1.edge-stack.v1 helm.sh/release.v1 1 14m
Versions
- Kubernetes - EKS (AWS) 1.21
- Chart version: 7.3.2
- Edge Stack Version: 2.2.2
Other Notes
- When I remove the licenseKey values passed in, the chart installs properly and the pods show up in a running state but it doesn't use our license key.
- In our automation we're inserting the license key in a secret before the helm chart is run, which is why we need to be able to use these in the values passed into the chart.
So I think I know the problem:
When we try to tell the helm chart which secret to reference, it's not actually using the "secretName" value.
licenseKey:
value: license-key
createSecret: false
secretName: edge-stack-license
So we pass in secretName: edge-stack-license and it actually is trying to find a secret called edge-stack.
Or in other words, its looking for a secretName that is the same name as the helm release name that was passed into the helm install.
Somehow this logic is not working:
volumesRaw: |
- name: {{ include "ambassador.fullname" . }}-secrets
secret:
{{- if and .Values.licenseKey .Values.licenseKey.secretName }}
secretName: {{ .Values.licenseKey.secretName }}
{{- else }}
secretName: {{ include "ambassador.fullname" . }}
{{- end }}
I'm passing in ".Values.licenseKey.secretName" yet it uses the "ambassador.fullname" instead.
https://github.com/datawire/edge-stack/blob/main/charts/edge-stack/values.yaml#L55
@LanceEa: any update on this? I am facing the same issue when using this Helm chart. We would like to save the API key in a Kubernetes Secret object, since there are some RBAC benefits to it.
