PICCL icon indicating copy to clipboard operation
PICCL copied to clipboard

Pipeline is slower for files which are combined (input files with same prefix)

Open peterdekker opened this issue 7 years ago • 0 comments

When running the pipeline for input files with the same prefix, the files are combined to one output file. Now I am testing on many-core hardware, it becomes apparent that combining files makes the pipeline much slower, especially in the OCR step. Probably this happens because parallelization across multiple CPU cores cannot be applied.

This is not a problem in itself, but I think it is good to notify users that files with the same prefix will be combined, and will have longer processing time. Or give them a choice to enable/disable combination. Now, a small difference in input, gives a large difference in processing time.

EDIT: I saw there is a "Reassamble PDF" option in the webinterface, but same-prefix files were also combined if I disabled this option.

peterdekker avatar Jun 12 '18 12:06 peterdekker