Simple chunking with nested parallelism

[rrichmond]

Hello,

I'm trying to get nested parallelism and some sort of chunking to work in a simple way with doFuture and %dopar%. Here's a simple example:

library(doFuture)
library(future.batchtools)

registerDoFuture()

plan(list(tweak(batchtools_slurm,
                template = "./slurm-simple.tmpl",
                resources = list(ncpus = 25,
                                 memory = "48GB",
                                 walltime="2:00:00",
                                 nodes=1)),
          multicore),
     workers=4)

## I want to loop over allvals and have the code run on the 4 worker, each in parallel across the 25 cores. The following works:

allvals <- 1:100

## This is custom chunking, but doesn't scale well and requires me to customize every time I change the number of values I want to loop over
out <- foreach(idx=1:4) %dopar% {
    outinner <- foreach(idx2=1:25) %dopar% {
        Sys.sleep(120)
        idx*idx2
    }
}

## What I really want is simple chunking of the entirety of allvals, but for the multicore inner plan to work. Something like the following would be great to be able to do and to have load balancing across the 4x25 cores:

out <- foreach(idx=allvals) %dopar% {
        Sys.sleep(120)
        idx
}

The issue with this later version is that it only runs on a single core on each of the 4 workers. Is there a simple way to distribute each of the 100 values to the 4 workers (25 to each, order doesn't matter) and ensure that on each worker the jobs run in parallel across the 25 cores? I find myself running into this sort of problem often with my cluster jobs, but haven't found a good solution with doFuture.

Thanks!