connect icon indicating copy to clipboard operation
connect copied to clipboard

Alternative workflow processor behavior when order is used

Open advancedlogic opened this issue 4 years ago • 1 comments

Currently the workflow processor runs branches trying to infere the DAG automatically or using order. The behavior of order prioritizes parallelism over sequential so in [ [ A, B], [C] ], A and B are processed simultaneously and then C is executed. When the graph becomes complex, it's hard to keep track of every parallel processing and it would be easier to describe the order as a graph with nodes and edges. Considering the example in the documentation:


      /--> B -------------|--> D
     /                   /
A --|          /--> E --|
     \--> C --|          \
               \----------|--> F

described as: [ [ A ], [ B, C ], [ E ], [ D, F ] ]

could become a list of edges:

[ [ A, B], [A, C], [B, D], [C, E ], [ C, F ], [ E, F ] ]

advancedlogic avatar May 01 '21 21:05 advancedlogic

https://github.com/benthosdev/benthos/issues/2599 - possible overlap with this issue

As part of the PR that proposes a solution to the issue 2599 - the way the DAG is represented in Benthos Config has been altered to :

          B:
            dependency_list: ["A"]
            processors:
              ...

So each Node in the DAG has a list of it's direct dependents.

jem-davies avatar May 28 '24 22:05 jem-davies