druid icon indicating copy to clipboard operation
druid copied to clipboard

Track and emit segment loading rate for HttpLoadQueuePeon on Coordinator

Open kfaraz opened this issue 1 year ago • 0 comments

Changes

  • Add LoadingRateTracker which computes a moving average load rate based on the last few GBs of successful segment loads.
  • Emit metric segment/loading/rateKbps from the Coordinator. In the future, we may also consider emitting this metric from the historicals themselves.
  • Add expectedLoadTimeMillis to response of API /druid/coordinator/v1/loadQueue?simple

Testing done

  • Added new unit tests
  • Performed testing on local cluster with ~1M segments

Release note

  • Emit new metric segment/loading/rateKbps from the Coordinator which tracks the current segment loading rate of a server. Dimensions: server
  • Add expectedLoadTimeMillis to response of API /druid/coordinator/v1/loadQueue?simple

Future work

The expectedLoadTimeMillis can be used to show expected time on the web-console.

This PR does not include web-console changes and the screenshot below is only for reference.

load_time

This PR has:

  • [x] been self-reviewed.
    • [ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
  • [x] added documentation for new or modified features or behaviors.
  • [x] a release note entry in the PR description.
  • [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • [ ] added or updated version, license, or notice information in licenses.yaml
  • [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • [ ] added integration tests.
  • [x] been tested in a test Druid cluster.

kfaraz avatar Jul 04 '24 05:07 kfaraz