redun Memory limits for local executor

I have noticed that redun seems to reserve memory even in local executors. Is there a way to tune that? or I am wrong on this assumption?

Jun 27 '22 14:06 jshleap

Hi @jshleap! Thanks for the question. I don't believe we set memory limits on local executors. Are you getting some behavior that suggests otherwise?

reserve memory even in local executors

Unless by "reserve" you mean preallocating memory. redun isn't doing anything unusual in that regard more than what multiprocessing thread and process pools typically do (FWIW, that is how redun implements the local executors). Let me know if that helps, or if I misunderstood the question.

Jun 27 '22 17:06 mattrasmus

Thanks for the answer @mattrasmus. I assumed that there is pre-allocation of memory per task as I am running out of memory for a medium sized dataset. I guess I should control the max_workers instead.

Jun 27 '22 17:06 jshleap

max_workers will work to limit parallelism across all tasks. If its just one task that should have parallelism limits, check out the limits config option: https://insitro.github.io/redun/config.html?highlight=limit#limits

Jun 27 '22 17:06 mattrasmus