Distributed cellpose using Dask for large (3D) images

[GFleishman]

Hi Cellpose Team!

This is quite late, but per discussions here: https://github.com/MouseLand/cellpose/issues/244 This is the distributed Cellpose that I am using for large 3D volumes (~2K x 6K x 5K voxels, maybe 150K total cell segments).

I've kept all new code in one place: cellpose/distribute.py The user should only be interacting with cellpose.distribute.distributed_eval I've written a not-so-complete-but-hopefully-helpful docstring for it. And here is an example of it being used on our LSF cluster:

distribued_cellpose_example.pdf

There are of course also some new dependencies (dask, and my wrapper for working with dask on a cluster). So setup.py and environment.yml also have some new additions.

I see that @chrisroat has also merged a solution. Maybe we can compare experiences? I'm happy to chat here or zoom.

Thanks, Greg

[carsen-stringer]

thanks @GFleishman! since this works on zarr arrays maybe you can add it to the contrib folder with the name distributed_zarr.py? also can you please put the dependencies in the setup.py under distributed? it would be great if you could add the example notebook as well, particularly if you have an example that can run proof of concept without a cluster

[GFleishman]

Happy to share it! I can make the suggested changes, but will do it after the symposium.

[lacan]

Is there an update to this getting merged?

[GFleishman]

New updated PR submitted where I actually follow through with the requirements for a merge.