Graceful shutdown of workers during kubernetes deploy

[knrafto]

If a deploy happens during a job, it will just be left hanging when the worker is killed. Instead, kubernetes should send a signal to the worker requesting a warm shut down, and the worker can exit gracefully after its current job.

See https://kubernetes.io/docs/user-guide/pods/#termination-of-pods. We need to:

• Update terminationGracePeriodSeconds in the deployment to a larger number (probably an hour or so) - see https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html
• Ensure the worker process receives the SIGTERM, and executes the RQ warm shut down

[Sumukh]

Adding this to the spec with a grace period should do the trick

        lifecycle:
          preStop:
            exec:
              # SIGTERM triggers a quick exit; gracefully terminate instead
              command: ["rq","suspend"]

Source: https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html