Support for multi-step jobs; i.e. run_settings as a list

[FoamScience]

Long term goal: Support for this

rb = exp.create_run_settings(exe="bin1")
rs = exp.create_run_settings(exe="bin2")

# Study parameters
params = {
    "nu": [1e-3, 1e-5, 1e-6],
}

ensemble = exp.create_ensemble("study", 
                               params=params, 
                               run_settings=[rb, rs],
                               perm_strategy="all_perm")

which runs bin1 and bin2 consecutively on each study case

Current state:

• Implemented only for local runners (but not on batch scripts)
• Some unit tests are on the way

[FoamScience]

Hey, @ashao Been a while; only got some time now for this - so I tried to push things a little bit.

Progress so far:

• Logging is better now; outputting to different err/out files for different job steps (currently, distinguished with the step's name, which is hash-like)

• Async execution of the steps makes things a bit complicated to manage; so I opted for the simplest solution:

  • Original behavior is always retained
  • If a list of run_settings is passed, only the last task is ran in async mode. We wait for the previous ones to complete.

  The premise of this PR is to be able to run something like this:

rb = exp.create_run_settings(exe="createMesh")
rs = exp.create_run_settings(exe="decomposeMesh")
rm = exp.create_run_settings(exe="solver", exe_args="-parallel", run_command="mpirun")
rm.set_tasks(4)

# Need to execute in this order; and wait for first two,
# because these commands depend on each other: rb ->  rs -> rm

# Study parameters
params = {
    "nu": [1e-3, 1e-5, 1e-6],
}

ensemble = exp.create_ensemble("study", 
                               params=params, 
                               run_settings=[rb, rs, rm],
                               perm_strategy="all_perm")

Please have a look; if you like this path, we can discuss how to improve it further

[FoamScience]

Just crossed my mind that letting users choose between async/sync tasks within job steps is actually useful. For example, if one runs a parameter variation to study the performance of the binaries, running them in async mode wouldn't make sense (they'll interfere with each other)