dask-cloudprovider icon indicating copy to clipboard operation
dask-cloudprovider copied to clipboard

Option to Provide Task Definition ARNs to ECSCluster

Open WToma opened this issue 3 years ago • 0 comments

Aims to resolve #135.

This set of changes provides an option to provide task definitions to the ECSCluster rather than having the cluster create one on startup. This can help with the proliferation of task definition families on the AWS UI, and with providing various settings to the task definition that the cluster doesn't support directly.

This PR is somewhat a work-in-progress, I just wanted to get some feedback on the direction. The biggest open question is the approach to the whole log-based address discovery situation. Currently the package uses CloudWatch logs to find the address of the scheduler and the worker. It has been suggested to remove this approach, but there may be complications. (See https://github.com/dask/dask-cloudprovider/issues/313#issuecomment-1171513920.) In order to be able to read the logs, we have to know the location of the logs. Previously, since we were managing the log locations via the task definitions, this wasn't a big issue. But now the user can provide task definitions which either don't log, or they log in a different region, etc. So the approach is now more brittle. Related to this, I also introduced options in this PR to have separate log groups and prefixes for workers and schedulers.

The documentation still needs to be updated to explain the log config changes. If the general approach looks good, I'll do that before merging.

Note: first time contributor here, sorry if I missed anything w.r.t formatting, or PR hygiene, or anything else—please let me know, and I'll fix those things.

WToma avatar Jun 30 '22 20:06 WToma