ssadedin
Resource manager: ability to specify "commands" per "job" submission?
Some supercomputing clusters don't assign you a "SLOT" in the cluster aka a cpu, instead they assign you a whole computer with N CPUS.
When this happens, it would be helpful to be able to get bpipe to submit N "commands" in a single cmd.sh file and wrap those in a "parallel -jN" to fully utilize a given cluster node.
This sounds like a pretty useful / important capability for these kind of clusters. Just thinking about how to implement it though, it probably is not trivial - Bpipe would need to buffer up jobs to try to fill up a node and only send the actual job when it has enough commands to run together. But then it still has to be smart enough to know if the pipeline can't move forward so it should just send a partially filled node. I'd be curious if you know of any other tools that have this capability, it would be interesting to look at how they implement it.
Magetbrain https://github.com/CobraLab/MAGeTbrain has this, but it doesn't have the same level of dependency tracking as your work, it only handles it at the stage level.
The other toolchain I use which does something like this is https://github.com/Mouse-Imaging-Centre/pydpiper
It handles jobs by running an 'executor' on each node which communicates back to the host machine and accepts jobs, you can then control the number of jobs per executor.
This allows me to run it both on my local cluster (SGE, 1 slot = 1 cpu) and my supercomputer account (torque, 1 slot = 8 cpus)
Good news! I figured out an alternative solution to my problem.
I'm going to write a little wrapper script that accepts a list of files and splits off chunks of jobs into local bpipe runs on each cluster machine. It's a bit crude, but it allows me to take advantage of the huge number of CPUs available in my cluster.
One question; Do you know how bpipe might handle contention on commandlog.txt? In this scenario all the jobs would be ideally spawned in the same directory and would be writing to the same logfile.
I'll write it up anyways and see how it goes.
After testing, I have written a dumb wrapper to allow me to use my existing bpipe pipelines on a cluster where 1 slot = 8 cpus:
https://github.com/CobraLab/minc-bpipe-library/blob/master/bpipe-batch.sh
That's an interesting solution - thanks for posting it. I think we may need something like this to support AWS / EC2. Typically if you start up an instance with a decent amount of RAM you'll get more capacity than you need to execute just 1 single threaded job. So having a way to multiplex several jobs to one EC2 instance is also useful.
On Fri, Jun 19, 2015 at 12:30 AM, Gabriel A. Devenyi < [email protected]> wrote:
After testing, I have written a dumb wrapper to allow me to use my existing bpipe pipelines on a cluster where 1 slot = 8 cpus:
https://github.com/CobraLab/minc-bpipe-library/blob/master/bpipe-batch.sh
— Reply to this email directly or view it on GitHub https://github.com/ssadedin/bpipe/issues/124#issuecomment-113173376.
I should note also note that I think that this may handle my issues with memory and large numbers of files as noted in #97. I could easily adapt this on a per-file basis on my existing cluster and then bpipe would be "fire and forget" without need for a controller anymore.
I have some users now doing 800+ files and we're running into max memory on the control machine as a result of bpipe's memory footprint.
I'm unhappy to report that splitting up the bpipe runs into chunks like this doesn't solve the memory issue.
At least this narrows down the origin of the issue
bpipe does not like lots of files in it's output folder, it seems to cause excessive memory usage.
My next step will be to have separate output folders per bpipe invocation to see if that solves the problem.
Take a look at this stage:
https://bitbucket.org/drambaldi/bpipe_config/src/407427ee61900de3e057ed16c0cd3d9db2e91f35/modules/sample_dir_gfu.groovy?at=master
Typically I create an output dir for each sample.
The downstream stages use a branch variable to keep track of output dir.
Pipeline:
https://bitbucket.org/drambaldi/bpipe_config/src/407427ee61900de3e057ed16c0cd3d9db2e91f35/pipelines/dna_alignment/genome_align_project.groovy?at=master
The input files are passed with a json file.
Hope it helps
Il giorno ven 26 giu 2015 00:05 Gabriel A. Devenyi [email protected] ha scritto:
I'm unhappy to report that splitting up the bpipe runs into chunks like this doesn't solve the memory issue.
At least this narrows down the origin of the issue
bpipe does not like lots of files in it's output folder, it seems to cause excessive memory usage.
My next step will be to have separate output folders per bpipe invocation to see if that solves the problem.
— Reply to this email directly or view it on GitHub https://github.com/ssadedin/bpipe/issues/124#issuecomment-115414804.
I definitely need to do some profiling on this to figure out the memory constraint. It's odd because I routinely run with a lot more files than that. One thing that can confuse the issue is that when Java runs out of file handles it will report an "OutOfMemory" error. So do check that it's not actually file handles at the root of it.
Cheers,
Simon
On Fri, Jun 19, 2015 at 11:41 PM, Gabriel A. Devenyi < [email protected]> wrote:
I should note also note that I think that this may handle my issues with memory and large numbers of files as noted in #97 https://github.com/ssadedin/bpipe/issues/97. I could easily adapt this on a per-file basis on my existing cluster and then bpipe would be "fire and forget" without need for a controller anymore.
I have some users now doing 800+ files and we're running into max memory on the control machine as a result of bpipe's memory footprint.
— Reply to this email directly or view it on GitHub https://github.com/ssadedin/bpipe/issues/124#issuecomment-113517909.