dask-drmaa
dask-drmaa copied to clipboard
Closing all workers
In cases where there are lots of workers stop_workers is slow. This appears to be caused by closing each worker in a for-loop.
As the common case where multiple workers are submitted to be stopped is just submitting all workers to stop, it would be good to special case this and pass JOB_IDS_SESSION_ALL to DRMAA. This would bypass the need for the loop.
There are a few options:
- Detect if all workers were passed in (easy, maybe a little hacky)
- Make the
worker_idsparameter optional and the default value equal to all workers (easy, straightforward) - Integrate this into
close(easy, hacky without 1 or 2) - Background the task somehow using threads, dask.delayed, or similar (may need a few tricks depending)
Thoughts?
Should add we need to take PR ( https://github.com/dask/dask-drmaa/pull/85 ) into account. So this would affect scale_down in a similar way.
cc @d-v-b