moduleBinaries param cannot be set in main.nf file

[davidmasp]

Bug report

Coming from this question (https://github.com/nextflow-io/nextflow/discussions/5203)

Expected behavior and actual behavior

According to the documentation, see here, when using module binaries we should include the parameter nextflow.enable.moduleBinaries either in the main file or in the configuration file.

To enable this feature, enable the following flag in your pipeline script or configuration file:

Including the parameter in the config file works as expected, however, including it in the main.nf file results in the following error.

$ nextflow run main.nf
Nextflow 24.04.4 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.04.3

Launching `main.nf` [kickass_agnesi] DSL2 - revision: 8091327fbc

ERROR ~ No such variable: moduleBinaries

 -- Check script 'main.nf' at line: 3 or see '.nextflow.log' file for more details

Steps to reproduce the problem

I have created a fork of the nextflow-io/hello repository to test this, see below:

https://github.com/davidmasp/hello-modbins

See commit -> 3766dd9 for the expected behavior when parameter is set in the config file. See last commit -> 5c91c64 for breaking behavior when parameter is set in the main.nf file

Program output

$ nextflow run main.nf
Nextflow 24.04.4 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.04.3

Launching `main.nf` [kickass_agnesi] DSL2 - revision: 8091327fbc

ERROR ~ No such variable: moduleBinaries

 -- Check script 'main.nf' at line: 3 or see '.nextflow.log' file for more details

Environment

• Nextflow version: 24.04.3
• Java version: openjdk version "17.0.10" 2024-01-16
• Operating system: macos but also tested in linux
• Bash version: zsh 5.9 (x86_64-apple-darwin23.0)

Additional context

None

[pditommaso]

Thanks for reporting this. i've updated the docs to reflect the current implementation 91ad1d42adcbaeece9219db7fb2fc1c74eaac61e

[bentsherman]

I asked @jorgee to add support for this feature flag in the script since feature flags should generally not be in the config

[stevekm]

Thanks for reporting this. i've updated the docs to reflect the current implementation 91ad1d4

It appears that the docs are not yet updated with this change

https://www.nextflow.io/docs/latest/module.html#module-binaries

[stevekm]

I was hoping to get some further clarification on this, the docs describe this feature as

enable the following flag in your pipeline script

which exact script are we referring to? It seems that the top-level main.nf pipeline script would be the obvious one, but are we planning for this setting to be possible to enable within the module itself? So if I have my repo set up such as modules/some-module/main.nf, I would be able to just pre-pend this line nextflow.enable.moduleBinaries = true within the module itself to enable this feature?

I wanted to check because this seems like it would be the ideal preferred situation; if you are writing a module, its meant to be portable, and if its going to be using a feature that requires an extra Nextflow config to be enabled else it will break and crash your pipeline, it seems like you would want the module to be able to enable the feature itself. Otherwise you will have the situation where someone tries to re-use or install a random module only for it to break their pipeline because they did not have a-priori knowledge that they were also supposed to update their Nextflow config with this setting to make the module not break.

[bentsherman]

The moduleBinaries feature flag can now be set in the pipeline script as of 24.10

As a best practice it should be set in the main script. I think it will work if you set it in the module script, but feature flags are applied to the entire pipeline so I would say it's better to define them all in the main script for clarity.

[stevekm]

I agree that it makes more sense to configure it in the main pipeline script, but I am not sure how we are supposed to utilize the feature inside modules and then communicate to the end-user that if they want to use the module, they need to set this feature. AFAIK the common usage is for users to use nf-core tools to "install" the modules, or more often people just copy/paste the module file or code into their pipeline, not clear if there was some intended way to handle this case where the module user is not aware that they need to make this config update for the module to work. Maybe there is a better forum to discuss this concern as well.

[davidmasp]

That's great news thanks @bentsherman and @jorgee 👏

So if I write a pipeline that includes modules with binaries, in the main script I add the config line and then it will include all binaries in any module. It should be transparent to the end-user of the pipeline.