proof icon indicating copy to clipboard operation
proof copied to clipboard

Parallelize Analysis execution

Open onyxfish opened this issue 10 years ago • 1 comments

Formerly onyxfish/agate#189

onyxfish avatar Sep 01 '15 00:09 onyxfish

One potential way of doing this would be to use dask. The Analysis object could be used to create a dask graph, and then run with the multiprocessing or threading scheduler. I'd probably work this up by traversing the Analyis object and creating a dask graph internally, but it could also potentially be done using dask.imperative).

Note that while dask has special classes for numpy/pandas wrappers, the core is only standard lib (pip install dask) and the multiprocessing scheduler has a very small dependency set (dill and toolz).

jcrist avatar Sep 01 '15 01:09 jcrist

Maybe better to just use dask or other tools, than to add this to proof (which is deliberately simple, like other wireservice tools).

jpmckinney avatar Oct 23 '23 20:10 jpmckinney