multithreading not working way faster

Open giovannirescia opened this issue 5 years ago • 2 comments

Hi,

I just run these two commands:

fastp -l 80 -n 3 -w 2 -i fastqdata/file_R1.fastq.gz -I fastqdata/file_R2.fastq.gz -o fastp_output/file_R1_2w.gz -O fastp_output/file_R2_16w.fastp_2w.gz

fastp -l 80 -n 3 -w 16 -i fastqdata/file_R1.fastq.gz -I fastqdata/file_R2.fastq.gz -o fastp_output/file_R1_16w.gz -O fastp_outputfile_R2_16w.fastp_16w.gz

And the execution times were:

w = 2: 1926 seconds
w = 16: 2185 seconds (first run), 1558 seconds (second run)

Is this somehow expected?

Server information: CORES: 56 RAM: 380 GB FILES SIZE: 15GB

Jul 06 '20 16:07 giovannirescia

make sure that I/O is not the bottleneck. NFS or heavily loaded file systems may reduce I/O performance drastically.

Jul 24 '20 08:07 sklages

Hi,

I have recently started using fastp and currently exploring the performance scalability of this tool. I think this issue is a relevant point to observe, but benchmark data would nice if it's already published somewhere. Thanks in advance.

Dec 01 '20 20:12 umeshwww