Feature Request: Allow the Use of Local Secrets with AWS Batch in Nextflow

[ShahzebMahmood]

Feature Request: Allowing the Use of Local Secrets with AWS Batch in Nextflow

Summary

A Feature request from a Seqera Platform user

Nextflow "secrets" not working with AWS Batch + Wave / Fusion from cli

Currently, Nextflow provides integration with AWS Batch, allowing users to execute workflows with secrets on Seqera Platform Which works as expected. However, there's a limitation when it comes to handling local secrets when baking them into a process and running it using the CLI.

Currently in the documentation: https://www.nextflow.io/docs/latest/secrets.html#process-directive

Proposed Solution

Integrate a feature into Nextflow that enables users to specify local secrets within there processes in their Nextflow scripts and run it via the CLI. These secrets would then be securely passed to AWS Batch jobs during execution.

[bentsherman]

These secrets would then be securely passed to AWS Batch jobs during execution.

This is the main challenge to my understanding. I'm not sure how the secrets could be passed securely, as they would be exposed as environment variables in the job definition.

@pditommaso to make this work I think we would need to extend the AWS Secrets Manager integration to work with Nextflow CLI runs and not just Seqera Platform.

[pditommaso]

This is by design, why using "local" secrets on batch?

[bentsherman]

I think the issue title is misleading, the actual requirement is to use secrets with AWS Batch for CLI runs. AWS secrets manager is only supported when using the platform but some customers want to leverage AWS secrets with CLI runs

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[robsyme]

Related: https://github.com/nextflow-io/nextflow/issues/5943