dask-jobqueue icon indicating copy to clipboard operation
dask-jobqueue copied to clipboard

CI: Clean up inconsistencies in the directory where the tests are run from

Open lesteve opened this issue 6 years ago • 0 comments

Left-over from #307. From https://github.com/dask/dask-jobqueue/pull/307#issuecomment-531300128:

it would be great if we could find a way not to have to cd in a different folder (jobqueue scheduler-specific, mostly PBS and SLURM). In an ideal world the /dask-jobqueue volumes would be mounted on the worker images I think but also pbsuser would have the right to write into /dask-jobqueue. Not sure how easy it is to do the second part in docker

❯ ag 'docker.+pytest' ci/
ci/slurm.sh
21:    docker exec -it slurmctld /bin/bash -c "pytest /dask-jobqueue/dask_jobqueue --verbose -E slurm -s"

ci/pbs.sh
22:    docker exec -it -u pbsuser pbs_master /bin/bash -c "cd; pytest /dask-jobqueue/dask_jobqueue --verbose -s -E pbs"

ci/sge.sh
20:    docker exec -it sge_master /bin/bash -c "cd /dask-jobqueue; pytest dask_jobqueue --verbose -s -E sge"
  • SLURM: we stay in / (i.e. no cd). Reason from https://github.com/dask/dask-jobqueue/pull/307#issuecomment-526645923

if you run sbatch /tmp/test.sh from a folder that does not exist in the worker node (e.g. /dask-jobqueue), the job terminates instantly. Somehow this seems related to the SLURM logging mechanism because if I set -e and -o to a folder that exists on the worker node the job runs fine. Maybe fixable in the docker images of the SLURM slaves.

  • PBS: we run as pbsuser so we have to cd somewhere where we can write (logs + local dask files). Here we cd into the pbsuser HOME directory. Reminder: you need to be root to write in /dask-jobqueue. Maybe fixable inside docker.
  • SGE: this was the default way of running the tests before #307.

lesteve avatar Dec 01 '19 16:12 lesteve