tasktiger Worker concurrency

Currently, a TaskTiger worker can process only one task concurrently. Is it possible to have one worker prefork multiple pool subprocesses so that it could process multiple tasks at same time?

In our case, we need preload some big modules(long time to load and a significant amount of memory usage) in parent process for performance. Starting multiple workers consumes too much memory, and this problem could be fixed if TaskTiger support worker concurrency.

May 09 '17 07:05 dawncold

Thanks! This is a good suggestion but not currently planned. PRs are welcome though!

Have you considered using batch queues and optimizing your task processing function so a single worker can process tasks quicker?

Sep 29 '17 17:09 thomasst

Thanks, batch queues should mitigate our case

Oct 09 '17 06:10 dawncold

@thomasst would like to work on this, any suggestion on api design you might want for this - i mean how do we mention concurrency on workers. Also i was wondering for workers might also make sense to allow thread level concurrency ie instead of preforking processes, it should be possible to use greenlets/threads/sub process right?

Oct 31 '17 13:10 kgritesh

You'd probably want an option that denotes how many workers you'd want to have and then just fork when starting the main process. I'm not convinced yet we should have an option to provide greenlets/threads.

Oct 31 '17 17:10 thomasst

@kgritesh Also looking for the same worker process concurrency. Did you get started on a fork with the changed logic?

Jan 05 '20 17:01 piercefreeman