druid
druid copied to clipboard
Track and emit segment loading rate for HttpLoadQueuePeon on Coordinator
Changes
- Add
LoadingRateTrackerwhich computes a moving average load rate based on the last few GBs of successful segment loads. - Emit metric
segment/loading/rateKbpsfrom the Coordinator. In the future, we may also consider emitting this metric from the historicals themselves. - Add
expectedLoadTimeMillisto response of API/druid/coordinator/v1/loadQueue?simple
Testing done
- Added new unit tests
- Performed testing on local cluster with ~1M segments
Release note
- Emit new metric
segment/loading/rateKbpsfrom the Coordinator which tracks the current segment loading rate of a server. Dimensions:server - Add
expectedLoadTimeMillisto response of API/druid/coordinator/v1/loadQueue?simple
Future work
The expectedLoadTimeMillis can be used to show expected time on the web-console.
This PR does not include web-console changes and the screenshot below is only for reference.
This PR has:
- [x] been self-reviewed.
- [ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in licenses.yaml
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.