Kashif Faraz

## Motivation Segment loading is one of the more complicated pieces of Druid operations and is prone to failures and race conditions. Due to a lack of clean logs and...

Proposal: Test framework to simulate segment balancing

1

## Motivation This proposal is inspired by the original thoughts in https://github.com/apache/druid/issues/9087 From a point of view of testing, segment balancing poses the following challenges: - In practice, balancing is...

[Failure logs](https://api.travis-ci.com/v3/job/576509179/log.txt) ``` [WARNING] Flakes: [WARNING] org.apache.druid.indexing.common.task.CompactionTaskRunTest.testRunIndexAndCompactForSameSegmentAtTheSameTime[SEGMENT] [ERROR] Run 1: CompactionTaskRunTest.testRunIndexAndCompactForSameSegmentAtTheSameTime:1422 expected: but was: [INFO] Run 2: PASS ``` The same tests passed with jdk11 but fail intermittently with jdk15

Improve documentation around granularities

1

The possible types of granularities that can be used are only mentioned within the querying section here: https://druid.apache.org/docs/latest/querying/granularities.html It's possible to use the same set of granularities as segment granularity...

As discussed [in this comment](https://github.com/apache/druid/issues/12692#issuecomment-1170869825), there are some issues with the `LocalIntermediaryDataManager`, typically involving race conditions. This is mostly because the peons as well as the middle manager perform discovery,...

[Failing job](https://app.travis-ci.com/github/apache/druid/jobs/575725132) [Sample failure logs](https://api.travis-ci.com/v3/job/575539290/log.txt) Failure was encountered a couple of times in #12737

Fixes multiple items in #12881 ### Description This PR lays the ground work to allow load queue to safely have an unlimited number of items, and thus eventually phase out...

Test: Remove mock LoadQueuePeon from coordinator unit tests

1

### Changes - Simplify `DruidCluster.Builder` - Merge `BroadcastDistributionRuleSerdeTest` into `BroadcastDistributionRuleTest` - Refactor `UnloadUnusedSegmentsTest` and other tests to avoid mocking `DruidServer` and `LoadQueuePeon` objects. - Add abstract test class `CoordinatorBaseTest`. This...

### Changes - Add `LoadingRateTracker` which computes a moving average load rate based on the last few GBs of successful segment loads. - Emit metric `segment/loading/rateKbps` from the Coordinator. In...

### Note - __This PR does not pertain to the audit logging system__ used by Druid for auditing all major update actions such as run a task, update rules, update...