Workers graceful shutdown being sure that current executions are not truncated

[andreapairon]

Hi everyone,

I have Conductor workers running on Kubernetes pods. I would like to handle the graceful shutdown in this way:

1. stop worker polling (I achieved this by implementing the function paused() of the WorkerInterface)
2. before executing the sys.exit() to terminate the pod, I want to be sure that the current worker executions will be finished without being truncated after the SIGTERM (so all the processes associated to the TaskHandler end)
3. finally execute sys.exit() to terminate the pod

I noticed that in the TaskRunner class, when the worker is paused, None is returned to the function run_once, but the function run is in a while True loop and I didn't understand how can I be sure when a worker is stopped and finished performing his task,

Is it possible to achieve the behaviour described in the 2nd point? Thank you in advance.

[coderabhigupta]

@andreapairon What version of SDK you're running on?

[andreapairon]

@coderabhigupta the latest, version 1.1.1

[andreapairon]

Maybe the trick could be: the while True in the TaskRunner class could be replaced by while self.worker.paused() == False. In that way every process would exit from the loop when his current execution finish, right?