dask
Add support for AWS Batch
It seems like AWS Batch can handle a lot of the logistics of bringing up and down the cluster automatically given a Docker image. I'm curious if it would make sense to integrate this into cloudprovider in some way: https://docs.aws.amazon.com/batch/latest/userguide/multi-node-parallel-jobs.html
I would be supportive of this. It would be good to add a BatchCluster as another option.
I don't currently have time to work on this, but would be happy to review a PR.
I wonder if there are any lessons to be learned or integration opportunities with https://github.com/nrdg/cloudknot?
Thanks for highlighting this @hammer.
I'm not sure the two are trying to achieve the same goals. Cloudknot seems to be trying to execute functions remotely on Batch from within some Python code.
Dask would be attempting to bootstrap a distributed cluster on Batch and then run the function on the distributed cluster.
Hi! Haven't contributed to Dask before but I'd be willing to look into this issue. I'm guessing the idea is to write a class similar to the Fargate one in https://github.com/dask/dask-cloudprovider/blob/master/dask_cloudprovider/providers/aws/ecs.py#L1167 but targeting AWS Batch?
