simChef icon indicating copy to clipboard operation
simChef copied to clipboard

Document HPC workflows

Open jpdunc23 opened this issue 2 years ago • 2 comments

Create an example using multiple nodes on a Slurm cluster to run a simulation.

jpdunc23 avatar Jan 15 '24 21:01 jpdunc23

Thanks @jpdunc23 , that would be super helpful. I am currently trying to use future.batchtools with the LSF backend to run some simChef simulations on our HPC, and I am not really sure how to get it right, and if it is possible to asynchronously run the simulations, i.e. start them, and collect results later, without letting the controlling R session run over the whole time period.

danielinteractive avatar May 27 '24 10:05 danielinteractive

@danielinteractive Glad you're trying this out! I'll try to take another look at this sometime soon. In the meantime, if you get it working we would greatly appreciate a PR extending the parallelization vignette or a new vignette altogether.

On the other hand, if you just want to get your simulations working as quickly as possible then I suggest dividing the simulation up into logical chunks (e.g., subsets of Method) if possible and run separate single-node LSF jobs for each subset. Please let us know how it goes one way or another.

jpdunc23 avatar May 29 '24 15:05 jpdunc23