[worker] gracefully terminate when a SIGTERM is received

[calebbrown]

The analysis worker in Kubernetes is sent a SIGTERM, and then 30 seconds later a SIGKILL.

I would be great if the analysis worker should gracefully stop working when it receives the SIGTERM.

Ideally this should be done with an algorithm like the following:

• don't start any new work if we've just finished
• wait a grace period (e.g. 5-10 seconds) to see if the current work finishes successfully
• use Go's context support to cancel a context after this grace period - if the analysis detects the context is cancelled it should immediately tear down the sandbox and abort execution without storing a result

[oliverchang]

Seems like a good first issue?

[calebbrown]

I think there are layers to this, some of which can be good for a first issue.

1. Not grabbing another message in the worker loop when SIGTERM has been received is easy.
2. Properly supporting Context in internal/analysis is probably moderately hard - but an experienced Go programmer may find it reasonably straight forward.
3. Wiring the context support up and adding a timer to trigger a context cancellation should be pretty easy.