[BUG] can't run `python -m distributed.cli.dask_spec`

[ntabris]

I'd like to be able to start Dask using python -m distributed.cli.dask_spec like so:

docker run rapidsai/base:24.04-cuda12.2-py3.9 python -m distributed.cli.dask_spec --spec [...]

But this gives me an error:

/opt/conda/bin/python: Error while finding module specification for 'distributed.cli.dask_spec.__main__' (ModuleNotFoundError: __path__ attribute not found on 'distributed.cli.dask_spec' while trying to find 'distributed.cli.dask_spec.__main__'); 'distributed.cli.dask_spec' is a package and cannot be directly executed

I don't understand what this means. Any idea why this isn't working? Can you make it work?

This seems to be something specific to the RAPIDS image, since this works fine with the normal dask container:

docker run daskdev/dask:latest python -m distributed.cli.dask_spec --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'

[rjzamora]

Thanks for raising this @ntabris ! This is indeed a bug.

I suggest we close the issue in favor of https://github.com/rapidsai/rapids-dask-dependency/issues/45, since the problem is caused by rapids-dask-dependency.

[ntabris]

Thanks @rjzamora. It would help me to know when this is fixed in the images, and using this issue to track that makes sense to me, but I don't strongly object if you want to close this (esp. if images would be pushed out with fix as soon as it's available).

[rjzamora]

It would help me to know when this is fixed in the images, and using this issue to track that makes sense to me

Okay, that seems fair to me.

Are you able to run dask spec instead of executing the distributed.cli.dask_spec module with python -m? I'd like to fix this problem for 24.06, but I'm not entirely sure if/when 24.04 can be addressed.

[ntabris]

Thanks for addressing this!

It would help me to know when this is fixed in the images, and using this issue to track that makes sense to me

Okay, that seems fair to me.

Alas, the bots disagree with my understanding of the distinctness of issues, and have closed this even though (I assume) there's no updated image yet.

Are you able to run dask spec instead of executing the distributed.cli.dask_spec module with python -m?

Maybe, but not easily.