Running gene_annotation.ipynb map_leafcutter_cluster_to_gene

[grennfp]

I was working on running gene_annotation.ipynb map_leafcutter_cluster_to_gene and had some issues getting the notebook to run. I commented on another issue which I thought could be related. The main issue seems to center around the job failing to create the tmp folder which the job later tries to use to run a python script (see the stderr file below).

I was able to fix this by updating my .bashrc file to include the export SINGULARITY_BIND=... line I have in my csg.yml file, followed by reloading it with the source command. Here are the relevant files before I got it to work:

Here is my stderr file:

/opt/conda/bin/python: can't open file '/restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/scripts/tmpdbgt_wz5/singularity_run_83472.py': [Errno 2] No such file or directory

My script:

#!/bin/bash -l

module load miniconda
conda activate xqtl_env2

sos run /restricted/projectnb/casa/frank/xqtl_project/xqtl-pipeline/pipeline/gene_annotation.ipynb map_leafcutter_cluster_to_gene \
    --cwd /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/leafcutter_output/batch_all \
    --intron_count  /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/leafcutter_output/batch_all/batch_all_bam_no_ext_intron_usage_perind.counts.gz  \
    --phenoFile /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/leafcutter_output/batch_all/batch_all_bam_no_ext_intron_usage_perind.counts.gz_raw_data.qqnorm.txt \
    --annotation-gtf /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.collapse_only.gene.ERCC.gtf \
    --container /restricted/projectnb/casa/frank/xqtl_project/xqtl-pipeline/container/singularity/bioinfo.sif \
    --walltime 100h \
    --numThreads 20 \
    --mem 40G -J 50 -c csg.yml -q neurology

And my job log file:

INFO: Running �[32mmap_leafcutter_cluster_to_gene�[0m: 
�[91mERROR�[0m: �[91mmap_leafcutter_cluster_to_gene (id=5a76d510fef6330f) returns an error.�[0m
�[91mERROR�[0m: �[91m[map_leafcutter_cluster_to_gene]: [0]: Executing script in Singularity returns an error (exitcode=2, stderr=/restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.collapse_only.gene.ERCC.gtf.stderr).
The script has been saved to /usr3/graduate/fgrennjr/.sos/4966833ed02ca234//usr3/graduate/fgrennjr/.sos/4966833ed02ca234.To reproduce the error please run:
�[0m�[32msingularity exec  /restricted/projectnb/casa/frank/xqtl_project/xqtl-pipeline/container/singularity/bioinfo.sif python /usr3/graduate/fgrennjr/.sos/4966833ed02ca234/singularity_run_83472.py�[0m�[91m�[0m