fiber icon indicating copy to clipboard operation
fiber copied to clipboard

fiber.Pool runs tasks in serial in local mode

Open coldfire913 opened this issue 4 years ago • 0 comments

Hi,

I am trying fiber v0.2.1 on local machine. the experiment runs the same task on 4 processes. the task takes approx. 68 secs. while multiprocessing lib works as expected, fiber looks like to be running in serial.

.fiberconfig [default] log_level=info log_file=stdout backend=local

def process(mplib, problem, num_tasks): pool = mplib.Pool(processes=4) futs = [ pool.apply_async(simple_solve, kwds={"problem": problem}) for _ in range(num_tasks) ] return [fut.get() for fut in futs]

res, elapsed = timeit(process, mp, problem, num_tasks) res2, elapsed2 = timeit(process, fiber, problem, num_tasks)

print("multiprocessing takes {} secs, results = {}".format(elapsed, res)) print("fiber takes {} secs, results = {}".format(elapsed2, res2))

multiprocessing takes 92.11180663108826 secs, results = [{'pid': 27242, }, {'pid': 27240, }, {'pid': 27241, }, {'pid': 27243, }] fiber takes 255.9156596660614 secs, results = [{'pid': 27265, }, {'pid': 27265, }, {'pid': 27265, }, {'pid': 27265, }]

fiber is always using the same process pid 27265 either push or pull queue, while the multiprocessing lib is distributing the works to 4 worker processes. but fiber's parzen_estimation example is working as expected on local machine.

coldfire913 avatar Jun 16 '21 09:06 coldfire913