Switch worker threads to have a worker hold thread, instead of a thread holding the worker

[jaggederest]

Currently we don't hold handles for the workers, we only hold their thread. This makes it difficult to gracefully shut down the workers when e.g. exiting a shortlived process or lambda function

[jaggederest]

This may be useful in the context of #1240 but there may also be other ways to solve that issue