go-vela
Multiple/concurrent worker processes
Description
Execute multiple worker process in concurrently. This will help execute multiple pipeline to manage many clusters/jobs at the same time.
Value
We manage around 15 Non-prod clusters and 14 Prod clusters. Any changes on the source code or configs will be validated across all the cluster at same time. Each jobs takes 5-15 minutes to validate the changes through pipeline. Running this in sequential order will be a time intensive process and will impact the delivery timeline of products. The concurrent worker process will help us speed this up.
Other Info
We have trying around some alternate approach and this process still takes around 3-7m for each pipeline. Considering 25+ clusters and running sequential takes around 2-4 hrs. This has high impact during production/development window. Also our cluster count is keep growing and running them without concurrent worker process going to be big challenge for us.
Added context: the introduction of stages seemed to solve a lot of the issues mentioned above. However, other teams came forward with concerns about overall performance, wanting quicker builds, etc.