Helm install --wait is not correct to pod

[dayeguilaiye]

When I run command helm upgrade test-release /path/to/my/chart --wait

I found that the wait is not correct when there is a pod in templates.

Reproduce:

1. Create a simple chart with a pod in it.
2. Install the chart (successfully).
3. Modify the image of pod in chart into a not exist image.
4. Upgrade the chart with --wait flag.
5. Now the wait should execute until timeout, but in fact, it pass through and return success instead. Expect: pod is not ready, release is waiting/failed. In fact: pod is not ready, release is deployed.

[gjenkins8]

https://github.com/helm/helm/blob/32f3691c3e0c924dcf4e3586adde869cb9d2d33f/pkg/kube/ready.go#L230-L238

Can you provide an example please. Helm should wait for pods like the above

[dayeguilaiye]

@gjenkins8 Yes, I found the code too, and the code is the reason why this problem happened.

After send patch request to k8s, helm will check isPodReady immediately. But the process of modifying the state is asynchronous with the process of making a request. As a result, there is a significant likelihood that the state may not be updated in time after the request returns, or even until the next query request arrives. So when helm check isPodReady, it's still PodReady. So helm will return successfully. But in fact, the pod might be failed after this update.

Here is example: I create a simple chart which only contains a simple pod like:

apiVersion: v1
kind: Pod
metadata:
  name: aaa
  namespace: default
spec:
  containers:
  - image: my/repository/test-image:v1
    imagePullPolicy: IfNotPresent
    name: aaa

The image my/repository/test-image:v1 is an exist image in my repository.

Then I run helm install test /path/to/my/chart --wait, then the command wait for few seconds, then succeed.

Every thing is right till now.

Then I change the image into my/repository/test-image:v100 which is not exist in my repository.

Then I run helm upgrade test /path/to/my/chart --wait. It return with success immediately:

Release "test" has been upgraded. Happy Helming!
NAME: test
LAST DEPLOYED: Wed Mar 20 10:06:49 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

but the pod is NOT READY:

[me:/home/ubuntu/test-chart]$ kubectl get pod
NAME                              READY   STATUS         RESTARTS     AGE
aaa                               0/1     ErrImagePull   0 (8s ago)   44s

[github-actions[bot]]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[gjenkins8]

marked as a bug, based on the above (but didn't get a chance to confirm/reproduce myself)

[antoineozenne-at-leocare]

Same issue here. It's a problem for us because we need to use --wait to notify the result of our deployment jobs to our developers.

[dayeguilaiye]

@antoineozenne-at-leocare helm can wait for deployment and jobs correctly. If you want helm to wait for deployment, you should run with --wait flag. But if you want it wait for jobs, you should run with both --wait and --wait-for-jobs flags.

This issue is about helm not waiting for pod.