How scheduler divides jobs between nodes?

Open orwel1984 opened this issue 8 years ago • 2 comments

So I have two nodes with 4 and 8 cpus.

I used the sample.py code to send 4 jobs to the cluster, in a for-loop using:

job = cluster.submit(random.randint(5,20))

what i see is that the schedular sends all the jobs to the first node with 4 CPUs, and the other node remains completely idle.

Is there a way to tell the schedular to divide the jobs equally between the nodes?? or control in a much more custom way?

Dec 16 '17 20:12 orwel1984

If you submit just 4 jobs, by the time second node is found and initialized, it is likely all jobs are already sent to the first node. Either delay submission until second node is also found, or submit more than 4 jobs so remaining jobs are held in scheduler for a bit.

Dec 16 '17 20:12 pgiri

so there is no way to configure that don't assign a particular node more than an x no. of jobs ?

Dec 17 '17 00:12 orwel1984