GLS support for mounting buckets with gcsfuse

[sereeena]

New feature

Running gcsfuse in a docker container to mount a gs bucket requires the container either to be run with --privileged, or with --device=/dev/fuse --cap-add=SYS_ADMIN for it to have access to the FUSE device.

I have tried specifying docker.runOptions in my nextflow.config file (with process executor google-lifesciences), and I have a docker container with gcsfuse installed, but when my script running in my docker container calls gcsfuse, I still get the error
/bin/fusermount: fuse device not found, try 'modprobe fuse' first mountWithArgs: mountWithConn: Mount: mount: running /bin/fusermount: exit status 1

I think the proper way to do this (and is what dsub does) is to pass the flag enable_fuse to the GLS action. I can see in the nextflow GoogleLifeSciencesHelper.groovy code that there is no way currently to pass an enable_fuse (or any) flag to the main action.

Usage scenario

Many people running life sciences data will often have multiple large reference files in cloud storage which they do not want to have to copy to their VMs before running their pipelines, it is much more convenient to be able to access a folder which mounts their cloud bucket, which is what gcsfuse does.

Suggest implementation

Perhaps add a new boolean field to GoogleLifeSciencesConfig for enable_fuse? Then in createMainAction it would check for this and pass the enable_fuse flag instead of an empty list.

I'm not sure how that would map to the nextflow.config file structure, because it possibly should be defined with the process and not the default google lifesciences config?

Just a question, if this was implemented would it also work for singularity containers?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[MatteoSchiavinato]

I recently found myself in dire need of this feature. Is there any step forward for it?

[pditommaso]

Google LS is a deprecated API, on the other hand Google Batch has a built-in support for it