Errors while running xpore dataprep

[JBerthelier]

Dear GoekeLab,

I am trying to run xpore on the cluster of our institute, everythings goes well using the demo data, however I got this error/warning while running xpore dataprep with my own data, by chance do you have any ideas of the causes and how to fix it ?

---

Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)/home/mycomputer/.local/ lib/python3.7/site-packages/xpore-2.1-py3.7.egg/xpore/scripts/dataprep.py:21: PerformanceWarning: indexing past lexsort depth may impact performance. pos_end += eventalign_result.loc[index]['line_length'].sum() /home/mycomputer/.local/lib/python3.7/site-packages/xpore-2.1-py3.7.egg/xpore/scripts/dataprep.py:72: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

---

Best regards,

Jeremy

[yuukiiwa]

Hi Jeremy,

Thanks for reaching out! It will be great if you can provide the command you used for running xpore dataprep. Other than that, you can also look into the following two things:

1. After you see this error/warning, was xpore dataprep still generating the dataprep/data.json file (see whether it increases in size by ls -lh dataprep/data.json)? If yes, xpore dataprep is still running fine.
2. What is the value you put in for --n_processes? Is this value larger than your environment variable "NUMEXPR_MAX_THREADS"? If yes, you might want to either change --n_processes to a smaller value or increase the value of your environment variable "NUMEXPR_MAX_THREADS"

Best wishes, Yuk Kei

[erika-fukuhara]

Hi Yuk Kei,

I’m working with Jeremy on running xpore dataprep.

Here is the command I used for running xpore data prep:

xpore dataprep \
--eventalign “eventalign_Araport11_GTF_genes_transposons-col0.txt" \
--gtf_or_gff “Araport11_GTF_genes_transposons_final_xpore.sorted.gtf" \
--transcript_fasta “Araport11_GTF_genes_transposons.fa" \
--out_dir dataprep \
--genome

After seeing the error/warning, xpore dataprep only generated the eventalign.index file. No other output files are generated when I try to run xpore dataprep.

Best, Erika

[yuukiiwa]

Hi Erika,

Thank you for the information! Do you mind showing me the head of eventalign_Araport11_GTF_genes_transposons-col0.txt, Araport11_GTF_genes_transposons_final_xpore.sorted.gtf, and Araport11_GTF_genes_transposons.fa, please? I am suspecting that this might be due to a customized gtf file.

Thanks!

Best wishes, Yuk Kei

[erika-fukuhara]

Hi Yuk Kei,

Here is the head for the eventalign.txt, GTF, and FASTA files.

eventalign_Araport11_GTF_genes_transposons-col0.txt:

contig	position	reference_kmer	read_index	strand	event_index	event_level_mean	event_stdv	event_length	model_kmer	model_meamodel_stdv	standardized_level	start_idx	end_idx
AT1G01020.2	426	TTCTG	29	t	429	78.67	1.821	0.00664	TTCTG	79.59	2.07	-0.36	29062	29082
AT1G01020.2	426	TTCTG	29	t	430	82.91	1.990	0.00332	TTCTG	79.59	2.07	1.32	29052	29062
AT1G01020.2	427	TCTGA	29	t	431	95.35	1.866	0.00232	TCTGA	91.37	2.85	1.15	29045	29052
AT1G01020.2	427	TCTGA	29	t	432	99.25	1.877	0.00631	TCTGA	91.37	2.85	2.27	29026	29045
AT1G01020.2	427	TCTGA	29	t	433	94.57	2.016	0.00266	TCTGA	91.37	2.85	0.92	29018	29026
AT1G01020.2	427	TCTGA	29	t	434	98.04	1.761	0.00797	TCTGA	91.37	2.85	1.92	28994	29018
AT1G01020.2	428	CTGAT	29	t	435	122.09	3.429	0.00730	CTGAT	111.64	4.49	1.91	28972	28994
AT1G01020.2	428	CTGAT	29	t	436	117.08	2.426	0.00299	CTGAT	111.64	4.49	0.99	28963	28972
AT1G01020.2	429	TGATT	29	t	437	136.43	6.966	0.00266	TGATT	127.73	5.10	1.40	28955	28963

Araport11_GTF_genes_transposons_final_xpore.sorted.gtf:

1	Araport11	transcript	3631	5899	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	3631	3913	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	3996	4276	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	4486	4605	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	4706	5095	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	5174	5326	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	5439	5899	.	+	.	gene_id "AT1G01010"; transcript_id "AT1G01010.1";
1	Araport11	exon	6788	7069	.	-	.	gene_id "AT1G01020"; transcript_id "AT1G01020.2";
1	Araport11	exon	6788	7069	.	-	.	gene_id "AT1G01020"; transcript_id "AT1G01020.6";
1	Araport11	exon	6788	7069	.	-	.	gene_id "AT1G01020"; transcript_id "AT1G01020.1";

Araport11_GTF_genes_transposons.fa:

>AT1G01010.1
AAATTATTAGATATACCAAACCAGAGAAAACAAATACATAATCGGAGAAATACAGATTACAGAGAGCGAG
AGAGATCGACGGCGAAGCTCTTTACCCGGAAACCATTGAAATCGGACGGTTTAGTGAAAATGGAGGATCA
AGTTGGGTTTGGGTTCCGTCCGAACGACGAGGAGCTCGTTGGTCACTATCTCCGTAACAAAATCGAAGGA
AACACTAGCCGCGACGTTGAAGTAGCCATCAGCGAGGTCAACATCTGTAGCTACGATCCTTGGAACTTGC
GCTTCCAGTCAAAGTACAAATCGAGAGATGCTATGTGGTACTTCTTCTCTCGTAGAGAAAACAACAAAGG
GAATCGACAGAGCAGGACAACGGTTTCTGGTAAATGGAAGCTTACCGGAGAATCTGTTGAGGTCAAGGAC
CAGTGGGGATTTTGTAGTGAGGGCTTTCGTGGTAAGATTGGTCATAAAAGGGTTTTGGTGTTCCTCGATG
GAAGATACCCTGACAAAACCAAATCTGATTGGGTTATCCACGAGTTCCACTACGACCTCTTACCAGAACA
TCAGAGGACATATGTCATCTGCAGACTTGAGTACAAGGGTGATGATGCGGACATTCTATCTGCTTATGCA

Thank you, Erika

[yuukiiwa]

Hi Erika,

Thank you for sharing the eventalign.txt, GTF, and FASTA files! Those should be compatible with xpore dataprep.

I think you should look into the first line of the error message Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS", which contacting the cluster maintainers of your institute will help.

Thanks!

Best wishes, Yuk Kei

[jeffersmith]

hello Yuk Kei

I am also trying to use xpore dataprep and Encountered the same problem，the dataprep/eventalign.index is generating, but data.json , data.index, data.log and data.readcount is empty, I have no idea about it and may I ask for your help? The command I running xpore dataprep is

 xpore dataprep \
--eventalign data/${file}/nanopolish/eventalign.txt \
--gtf_or_gff all.gtf \
 --transcript_fasta ref.fa \
--out_dir data/${file}/dataprep \
--genome

I got error

/mycomputer/miniconda3/lib/python3.9/site-packages/xpore-2.1-py3.9.egg/xpore/scripts/dataprep.py:21: PerformanceWarning: indexing past lexsort depth may impact performance.
  pos_end += eventalign_result.loc[index]['line_length'].sum()
/mycomputer/miniconda3/lib/python3.9/site-packages/xpore-2.1-py3.9.egg/xpore/scripts/dataprep.py:72: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

And my eventalign.txt, GTF, and FASTA all seem like @erika-fukuhara, do you solve this problem or have any suggestion?

Thank you! Jeffer

[acarmas1]

Hey,

I'm having the same problem. I run xpore dataprep but the data.json data.log and other files are empty.

Do you know how we can fix it?