run nf-test with awsbatch executor

[fmalmeida]

Hy dev team, Thanks for this nice tool.

When trying to execute nf-test with awsbatch it failes with:

Nextflow stderr:
  
  When using `awsbatch` executor an S3 bucket must be provided as working directory using either the `-bucket-dir` or `-work-dir` command line option

even tough workDir is set to a S3 path in nf-test.config.

I saw that the S3 path actually was created locally by nf-test. So my question are:

• Is this a known limitation by design? If so, there are plans for integration?
• Or else, is there something wrong that nees fine-tuning?

💭

[seppinho]

Yes, that's currently a known limitation. We were discussing this internally today and we see the advantage of having this. We'll come back to you when we have a plan how to integrate that.

[blaurenczy]

Hi! I see that this question here about awsbatch is very much related to mine (here: https://github.com/askimed/nf-test/issues/91) about Azure Cloud, so I understand this feature has been considered but there is no plan yet?

[ivopieniak]

That would be fantastic feature! +1

[blazejszczerba]

Hey, any progress on that ?

[arondaniel]

Hello,

In your nf-test.config you can add: options "-bucket-dir s3://your-bucket/some/path"

This at least allows my test to run with an awsbatch profile. However any files returned in output channels will be at that s3 location.

[PikalaxALT]

+1 on this issue, it is critical for my ci/cd testing

[stevekm]

for some added context, you can see why this is happening if you add this line to your Nextflow main.nf

println("workflow.commandLine: ${workflow.commandLine}")

and then run nf-test test --verbose main.nf.test;

    > workflow.commandLine: nextflow -log /home/username/projects/my-pipeline/.nf-test/tests/3431ab1030a8e7bd96d6c6d85973429/meta/nextflow.log run /home/username/projects/my-pipeline/main.nf 
-c /home/username/projects/my-pipeline/nextflow.config 
-c /home/username/projects/my-pipeline/tests/nextflow.config 
-params-file /home/username/projects/my-pipeline/.nf-test/tests/3431ab1030a8e7bd96d6c6d85973429/meta/params.json 
-ansi-log false 
-profile awsbatch 
-with-trace /home/username/projects/my-pipeline/.nf-test/tests/3431ab1030a8e7bd96d6c6d85973429/meta/trace.csv 
-w /home/username/projects/my-pipeline/.nf-test/tests/3431ab1030a8e7bd96d6c6d85973429/work

You can see that nf-test is appending the Nextflow cli arg -w to configure the work dir to a local directory, despite having the config set as this inside my nextflow.config for the awsbatch profile;

workDir = 's3://my-bucket/nextflow/workDir'

Notably, neither of these possible solutions within nf-test seem to work;

nextflow_pipeline {
    script "main.nf"
    profile "awsbatch"
    workDir "s3://my-bucket/nextflow/workDir"

gives error

🚀 nf-test 0.9.2
https://www.nf-test.com
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr

Error: groovy.lang.MissingMethodException: No signature of method: main_nf$_run_closure1.workDir() is applicable for argument types: (String) values: [s3://my-bucket/nextflow/workDir"]

while this method

nextflow_pipeline {
    script "main.nf"
    profile "awsbatch"
    options "-work-dir s3://my-bucket/nextflow/workDir"

gives error

  Test [3431ab10] 'Should run without failures'
    > Nextflow 24.10.0 is available - Please consider updating your version to it
    > Can only specify option -w once. -- Check the available commands and options and syntax with 'help'
    FAILED (2.144s)

So unless I am missing something (?) it appears that there is no possible way to run nf-test but to execute the tasks on AWS Batch, as you normally can do trivially with Nextflow. Is this really the case? If so I would think this would be a serious issue because many pipelines' storage and compute requirements are too large to run locally even with a minimal dataset.

Let me know if there is some way to get AWS Batch execution working with Nextflow in nf-test, thanks

[harmonbhasin]

Another +1 on getting this feature implemented.

[just-simon-5]

I would also like to see this implemented!

[stevekm]

as a follow up, I see that this does work instead;

nextflow_pipeline {
    name "Test case"
    script "main.nf"
    profile "awsbatch"
    options "-bucket-dir s3://my-bucket/nextflow/workdir"