dask-jobqueue

dask-jobqueue copied to clipboard

dask-jobqueue for Fujitsu HPC

3

In the dask-jobqueue, there are a lot of APIs for supercomputer structures. However, there is no Fujitsu HPC. Fujitsu structure is widely used in Japan. In the job script, it...

MichaelChaoLi-cpu

Handshake timeout error on SGE cluster

7

``` from dask_jobqueue import SGECluster from dask.distributed import Client, as_completed import blm_setup, blm_batch, blm_concat, blm_cleanup # my own functions which don't employ dask # Given a dask distributed client run...

TomMaullin

Make job submission asynchronous

9

I have noticed that execution of commands (e.g., `condor_submit` for the condor backend) appear to be synchronous. In fact, there's a small note about this in the code itself: https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/core.py#L305...

bbockelm

Adaptive scaling and dask-jobqueue goes into endless loop when a job launches several worker processes (was: Different configs result in worker death)

17

**What happened**: (Reposting from [SO](https://stackoverflow.com/questions/67326553/dask-jobqueue-why-does-using-processes-result-in-cancelled-jobs/67328294?noredirect=1#comment119067830_67328294)) I'm using Dask Jobequeue on a Slurm supercomputer (I'll note that this is also a Cray machine). My workload includes a mix of threaded (i.e....

AlecThomson

Python entry is missing in config file

2

I don’t know if it is on purpose or not but `python` argument can’t be set from config file (if not provided at cluster initialization, it is set to `sys.executable`)....

PhML

[draft] Drop Python 3.7 and allow tests to work in isolation

2

graingert

I hesitated to put this issue as a comment in #567, but I'm under the impression we have two separate problems. In #562, I override the `cleanup` fixture from distributed...

guillaumeeb

See discussions in #565. In particular https://github.com/dask/dask-jobqueue/pull/565#pullrequestreview-1052591018. It is not safe to create a `ProcessInterface` without a running loop. Use of the default event loop is deprecated in Python. We...

guillaumeeb

tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x2ab42ba925d0>>, <Task finished coro=<Worker.heartbeat() done

16

I am trying to do data analysis on the 9900 parquet files that in total they have 100GB size. After 70K garbage collections warning: `distributed.utils_perf - WARNING - full garbage...

MSKazemi

Since https://github.com/dask/dask-jobqueue/pull/545, we have added in most of the CI runs a `docker compose build` command that was not there (by choice) to use pre-backed Docker images. Without a `docker...

guillaumeeb