python-task-queue icon indicating copy to clipboard operation
python-task-queue copied to clipboard

Task Dependencies

Open william-silversmith opened this issue 4 years ago • 0 comments

It would be useful to support task chaining. This allows processors to guarantee locality when acquiring the next task.

The simplest version of this would be to insert a list of tasks. A more sophisticated version would allow for the specification of a tree that would process breadth-first.

There are drawbacks to using this kind of processing mode. One of the big ones is that tasks with dependencies won't benefit from automatic parallelism and retries of individual tasks. Instead, the whole unit must complete or be retried.

william-silversmith avatar Apr 13 '21 16:04 william-silversmith