Auto-Compaction parent tasks not started if any are still running

[gl]

Affected Version

Druid 27

Description

We are running a very large Druid cluster and we have 8 autocompaction tasks ("compact" type) running in parallel to consolidate a kafka-based datasource. Although the compaction tasks don't have the same duration, we have noticed that they are always in sync, meaning that there is no new compact task started if there's at least one still running. This is technically ok, but it's a massively inefficient use of CPU resources.

For clarity, the compaction tasks duration is roughly 4-5 hours, and druid.coordinator.period.indexingPeriod=PT10M.

It doesn't look like there is any way to tune this, and the behaviour seems undocumented as far as I can read, so as it's not expected, I'm filling it as a PR.

[github-actions[bot]]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.