reach icon indicating copy to clipboard operation
reach copied to clipboard

Investigate paralysation of yielding structured references

Open lizgzil opened this issue 5 years ago • 0 comments

In split_reach/extracter/extract_refs_task.py we set pool_map = map for use in yield_structured_references. However if we utilise Pool from multiprocessing i.e.

pool = Pool(num_workers)
pool_map = pool.map

we could speed up this task. However in the past we found using num_workers>1 actually slowed things down. So worth investigating how this behaves now to see whether it's worth implementing.

lizgzil avatar Apr 30 '20 14:04 lizgzil