dysgu_updated

[poddarharsh15]

PR checklist

Closes #5784

• [ ] This comment contains a description of changes (with reason).
• [ ] If you've fixed a bug or added code that should be tested, add tests!
• [x] If you've added a new tool - have you followed the module conventions in the contribution docs
• [ ] If necessary, include test data in your PR.
• [x] Remove all TODO statements.
• [ ] Emit the versions.yml file.
• [ ] Follow the naming conventions.
• [x] Follow the parameters requirements.
• [ ] Follow the input/output options guidelines.
• [ ] Add a resource label
• [x] Use BioConda and BioContainers if possible to fulfil software requirements.
• Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
  • For modules:
    • [x] nf-core modules test <MODULE> --profile docker
    • [ ] nf-core modules test <MODULE> --profile singularity
    • [x] nf-core modules test <MODULE> --profile conda
  • For subworkflows:
    • [ ] nf-core subworkflows test <SUBWORKFLOW> --profile docker
    • [ ] nf-core subworkflows test <SUBWORKFLOW> --profile singularity
    • [ ] nf-core subworkflows test <SUBWORKFLOW> --profile conda

[poddarharsh15]

@famosab I have updated main.nf and meta.yml files for dysgu module please have a look, Thanks :)

[famosab]

I added some suggestions to include also gzipped files and also emit the index to the gzipped vcf (I think that is best practise). Maybe you can apply those changes and also run prettier on the code so that the tests might run.

[poddarharsh15]

Great, thanks I will accept all the commits and run prettier asap :)

[poddarharsh15]

I added some suggestions to include also gzipped files and also emit the index to the gzipped vcf (I think that is best practise). Maybe you can apply those changes and also run prettier on the code so that the tests might run.

Hi @famosab I tried to run prettier tests on github but it's emitting several errors, do I need to install prettier --check on my workstation to solve the errors?

[famosab]

I added some suggestions to include also gzipped files and also emit the index to the gzipped vcf (I think that is best practise). Maybe you can apply those changes and also run prettier on the code so that the tests might run.

Hi @famosab I tried to run prettier tests on github but it's emitting several errors, do I need to install prettier --check on my workstation to solve the errors?

You will need to use prettier to format your code on your machine and then push the changes. Other approaches did not work for me for some reason:

conda install prettier

Once installed, you can run prettier manually:

prettier -w .

This will overwrite files with formatting fixes To just check without auto-fixing, use prettier -c .

[poddarharsh15]

I added some suggestions to include also gzipped files and also emit the index to the gzipped vcf (I think that is best practise). Maybe you can apply those changes and also run prettier on the code so that the tests might run.

Hi @famosab I tried to run prettier tests on github but it's emitting several errors, do I need to install prettier --check on my workstation to solve the errors?

You will need to use prettier to format your code on your machine and then push the changes. Other approaches did not work for me for some reason:

conda install prettier

Once installed, you can run prettier manually:

prettier -w .

This will overwrite files with formatting fixes To just check without auto-fixing, use prettier -c .

Okay, I will do that. One more concern: is it necessary to finish the main.nf.test before running prettier --check? Thank you very much.

[famosab]

No you dont need that.

prettier -w . formats the code so that its adhering to the standards.

main.nf.test are the tests written with nf-test which will ensure that your module works as expected. I would first use prettier to format the code and then we can tackle how to write the test.

[famosab]

For the main.nf.test file: have a look at other modules that are already ported to nf-test. You will always need to use data from the test data repository and then assertions to check if the module runs correctly. I implemented some tests for strelka. But you can also check any other module.

[poddarharsh15]

No you dont need that.

prettier -w . formats the code so that its adhering to the standards.

main.nf.test are the tests written with nf-test which will ensure that your module works as expected. I would first use prettier to format the code and then we can tackle how to write the test.

I have successfully installed prettier on workstation now I will try to run prettier -w on main.nf first.

[famosab]

For the editorconfig test: https://nf-co.re/events/2023/bytesize_precommit

[poddarharsh15]

For the editorconfig test: https://nf-co.re/events/2023/bytesize_precommit

Hi @famosab I used editor.config in VScode but I am still getting some errors with wrong amount of left-padding spaces(want multiple of 4) Could you please have a look at it, Thanks.

[famosab]

@poddarharsh15 its solved. Now you'll need to propely edit the main.nf.test file and then run nf-core modules test dysgu on your machine to create the main.nf.test.snap file (so that the linting test passes). Note that you'll need to run docker :)

[poddarharsh15]

@poddarharsh15 its solved. Now you'll need to propely edit the main.nf.test file and then run nf-core modules test dysgu on your machine to create the main.nf.test.snap file (so that the linting test passes). Note that you'll need to run docker :)

Thanks again @famosab, I have started working on main.nf.test and, with the help of Strelka and other modules, I believe I will be able to finish it by tomorrow. I will update you as soon as I have run all the tests. I highly appreciate your help :)

[poddarharsh15]

Hi @famosab I am getting this error after accepting the changes in main.nf.test please find the reference attached with this message. Thanks nextflow.log

[famosab]

I might have introduced some kind of syntax error. I'll clone it and check.

[famosab]

The tests do fail for me, but I noticed that its cram files and we probably need bams?

[poddarharsh15]

The tests do fail for me, but I noticed that its cram files and we probably need bams?

In theory, Dysgu works with both BAM and CRAM files, but for now, we can try running the tests with just BAM files. I apologize for any confusion caused by my previous comments. with these as input :-

input[0] = [ [ id:'test' ], // meta map file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam', checkIfExists: true), file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam.bai', checkIfExists: true) ]

[famosab]

Related to #5784

[famosab]

The tests do fail for me, but I noticed that its cram files and we probably need bams?

In theory, Dysgu works with both BAM and CRAM files, but for now, we can try running the tests with just BAM files. I apologize for any confusion caused by my previous comments. with these as input :-

input[0] = [ [ id:'test' ], // meta map file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam', checkIfExists: true), file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam.bai', checkIfExists: true) ]

We can introduce both tests that is no problem

[famosab]

@poddarharsh15 there is some issue with empty files - maybe the chaining in the output does not work as expected. But code skelton wise the things should work now.

[poddarharsh15]

@poddarharsh15 there is some issue with empty files - maybe the chaining in the output does not work as expected. But code skelton wise the things should work now.

@famosab I believe creating separate folders for each process of dysgu might lead to better results. It could be an issue with the dysgu command line, which I will review again soon, likely on Monday.

[famosab]

If the conda test stays unstable it might be worth trying different assertions as I shown above.

[poddarharsh15]

If the conda test stays unstable it might be worth trying different assertions as I shown above.

I accepted the assertions now I will give it a try with nf-core modules test dysgu I will keep you updated :)

[poddarharsh15]

Hello, @famosab I successfully created the main.nf.test.snap file, and one of the assertion tests ('human - bam - stub') passed. Could you please take a look at the errors that others failed?

[famosab]

Can you have a look at the resulting vcfs? Maybe dysgu reports version 4.1. Then just change that. You can find the vcfs in the corresponding folder given in the error. /home/tigem...

[poddarharsh15]

Can you have a look at the resulting vcfs? Maybe dysgu reports version 4.1. Then just change that. You can find the vcfs in the corresponding folder given in the error. /home/tigem...

I looked in the work directories and the vcf files are empty but I have checked cmd.err files and I think the errors are emitting because of --temp-directory

Unable to find image 'quay.io/biocontainers/dysgu:1.6.2--py310h770aed0_0' locally 1.6.2--py310h770aed0_0: Pulling from biocontainers/dysgu f03c707b3130: Already exists 5e3ed8cd934c: Already exists c8581de4638a: Already exists 24cbf6bc5826: Already exists bd9ddc54bea9: Already exists b7e552ecadde: Pulling fs layer b7e552ecadde: Download complete b7e552ecadde: Pull complete Digest: sha256:68e7414538c175ceddab05c5a8e402ca5e5cacb545a3d7f4ba4a67146a4d6ac5 Status: Downloaded newer image for quay.io/biocontainers/dysgu:1.6.2--py310h770aed0_0 Usage: dysgu run [OPTIONS] REFERENCE WORKING_DIRECTORY BAM Try 'dysgu run --help' for help.

Error: No such option: --temp-directory

command.log PS: I have checked previous result from dysgu-run (not with nf-core) and dysgu reports vcf version 4.2.

[famosab]

@poddarharsh15 maybe it makes sense to start another clean PR where no files are changed that should not be, you could just create a branch in your repo, implemented the changes we actually want and then open the PR

[poddarharsh15]

@poddarharsh15 maybe it makes sense to start another clean PR where no files are changed that should not be, you could just create a branch in your repo, implemented the changes we actually want and then open the PR

@famosab Sure, that makes sense. I'll create a new branch in my repo, implement the necessary changes, and then open a clean PR.[additionally, I will add run add call submodules], thanks for the update I will get back to you asap :)