Use ECS API to set Worker/Scheduler address instead of parsing logs

[anovv]

From https://github.com/dask/dask-cloudprovider/issues/47

ECSCluster parses logs to get Worker/Scheduler address, this may lead to bugs. We can use ECS API to get private/public IPs

https://github.com/dask/dask-cloudprovider/issues/47#issuecomment-931056815 https://github.com/dask/dask-cloudprovider/issues/47#issuecomment-932166175 https://github.com/dask/dask-cloudprovider/issues/47#issuecomment-932298592

[WToma]

I looked into this, and I think an extra piece of complication is the port assignment: for workers, which currently listen on a random port, using the logs is needed to find the random port they chose during startup. That said, I couldn't find if the worker address is actually needed directly by the Client or the Cluster. So maybe that could be removed.

[jacobtomlinson]

I don't think the worker address matters. The workers are given the scheduler address and then they register themselves with the scheduler. The client and cluster do not need this information.

The only time this is a challenge is when the address the worker thinks it is listening on is different to the address the scheduler needs to connect to. This can happen in containerised environments with overlay networking and port mappings, but the worker does have a contact_address config option where you can tell it explicitly what address it needs to give the scheduler.