Request for toil-cwl-runner: user provided mappings from requirements to queue names.

[DailyDreaming]

https://docs.google.com/document/d/1dQP2QfmJRmUtwuM5hKnB1rktM2DUpghdut-_fd5xELQ/edit?ts=5ef2415f#heading=h.aenwtc5w5k03

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-573

[ghost]

Thanks for opening the ticket @DailyDreaming . It is my opinion this should be a command line parameter passed to Toil for appropriate backends, instead of being hardcoded into a workflow.

[adamnovak]

The idea seems to be that we would have Kubernetes label selectors and/or HPC system distinct queue names tracked along with Toil jobs, and then have support for setting selectors or destination queues for jobs using some kind of semi-standard requirements in the CWL front-end, and then we'd have a way to configure what CWL special requirements correspond to what constraints in the backing batch system.

[adamnovak]

@DailyDreaming Did I describe this the way you remember?

[mberacochea]

A queue mapping procedure would be very useful. One common case is using different queues based on the memory required:

For example a mapping.yml file such as:

# first match wins, in the order presented here
bigmem: 
    memMin: 200G
standard:
    memMax: 10G