fix-5612 / s3 cloud-storage compatibility
PR checklist
Closes #5612
To address issue #5612, the following changes have been made to ensure compatibility with cloud storage like S3:
-
File Path Creation: Replaced the Java method
new File()with Nextflow'sfile()method to properly handle file paths in cloud storage environments. -
File Writing Method: Replaced direct Java I/O operations with Nextflow's
withWriterAppendfor appending to the log file, maintaining consistency with Nextflow. -
Type Change: Changed the parameter type from
FiletoPathin theappendToLogFilefunction to align with Nextflow.
Tests required to be performed in an s3 enviroment, samplesheet should have at least one bad barcode to test the entire functionality.
- [x] This comment contains a description of changes (with reason).
- [x] If you've fixed a bug or added code that should be tested, add tests!
- [ ] If you've added a new tool - have you followed the module conventions in the contribution docs
- [ ] If necessary, include test data in your PR.
- [ ] Remove all TODO statements.
- [ ] Emit the
versions.ymlfile. - [ ] Follow the naming conventions.
- [ ] Follow the parameters requirements.
- [ ] Follow the input/output options guidelines.
- [ ] Add a resource
label - [ ] Use BioConda and BioContainers if possible to fulfil software requirements.
- Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
- For modules:
- [ ]
nf-core modules test <MODULE> --profile docker - [ ]
nf-core modules test <MODULE> --profile singularity - [ ]
nf-core modules test <MODULE> --profile conda
- [ ]
- For subworkflows:
- [ ]
nf-core subworkflows test <SUBWORKFLOW> --profile docker - [ ]
nf-core subworkflows test <SUBWORKFLOW> --profile singularity - [ ]
nf-core subworkflows test <SUBWORKFLOW> --profile conda
- [ ]
- For modules:
Hey @matthdsm @k1sauce, any updates on this? I am getting this issue of emtpy fastqs quite a lot in our production, so I need to move forward with either of the fixes.
` executor > local (1) [3d/f5bc64] NFC…CLCONVERT (22F37MLT3.null) | 0 of 1 [- ] NFC…ULTIPLEX:DEMULTIPLEX:FALCO - [- ] NFC…LTIPLEX:DEMULTIPLEX:MD5SUM - [- ] NFC…USTOM_DUMPSOFTWAREVERSIONS - [- ] NFC…TIPLEX:DEMULTIPLEX:MULTIQC - ERROR ~ Cannot invoke method startsWith() on null object
-- Check script '/home/sylvesterportal/.nextflow/assets/nf-core/demultiplex/./workflows/../subworkflows/nf-core/bcl_demultiplex/main.nf' at line: 125 or see '.nextflow.log' file for more details`
@glichtenstein , can we close this PR now, given #5720 ?
This PR was a proposal to close #5612 by retaining the functionality to log the empty FASTQ files that will be skipped during the Falco step, logging them to a file for easy identification. Additionally, it included changes to ensure compatibility with S3 cloud storage. Given the refactoring in #5720, which handles FASTQ grouping differently, and the team’s preference to avoid creating a log file due to some S3 misbehavior and to prevent unexpected file creation for some users, we have opted to merge #5720 instead. Therefore, I am closing this PR in favor of #5720. Thank you to everyone involved.