distributed icon indicating copy to clipboard operation
distributed copied to clipboard

Add missing options to dask-ssh cli

Open abduhbm opened this issue 5 years ago • 4 comments

Closes #3763

This PR add missing options from dask-scheduler and dask-worker commands to dask-ssh.

Example:

$ dask-ssh --nprocs 2 --nthreads 1 --worker-port 9000:9010  --hostfile ../nodes \
--scheduler-dashboard-address :8989 --scheduler-preload /path/to/foo.py

abduhbm avatar May 06 '20 17:05 abduhbm

Thanks @jcrist for your feedback.

test_bokeh_deprecation keeps failing in CI checks. It passes on my local setup. Any idea why?

abduhbm avatar May 07 '20 20:05 abduhbm

test_bokeh_deprecation keeps failing in CI checks. It passes on my local setup. Any idea why?

Not sure, but since you've changed the CLI here I'd expect this to be due to your changes. Usually when I run into a test failing I do the following:

  • I find the name of the test, and use py.test locally run just that test:
# The `-k` flag tells pytest to run just that test
$ py.test distributed/cli/tests/test_dask_worker.py -k test_bokeh_deprecation
  • If looking at the failure doesn't help, I might also run py.test with --pdb, to drop into a debugger on failure
# The `--pdb` flag tells pytest to start a pdb session on error
$ py.test distributed/cli/tests/test_dask_worker.py -k test_bokeh_deprecation --pdb
  • If that still isn't useful, I might make a separate script (or in this case CLI command) to reproduce and iterate from there. If needed, I might add a breakpoint() call somewhere in the code where I want to stop and inspect what's going on. In this case, it looks like the --bokeh-port and --bokeh flags are possibly no longer setting the bokeh_port and bokeh parameters properly. You might start looking here: https://github.com/dask/distributed/blob/master/distributed/cli/dask_worker.py#L269-L279 aren't being forwarded properly.

If after looking through this you're still having trouble debugging, let me know and I'll take a closer look.

jcrist avatar May 11 '20 19:05 jcrist

@jcrist it seems that it was an import order issue for Click options. pytest runs test_dask_ssh before test_dask_worker and will import and register Click Options for distributed.cli.dask_ssh.main before distributed.cli.dask_worker.main.

I modified some cli tests to run them using subprocess.Popen instead of CliRunner.invoke to avoid this import issue.

abduhbm avatar May 11 '20 20:05 abduhbm

@jcrist please let me know if anything else is needed here. Thanks,

abduhbm avatar May 21 '20 13:05 abduhbm