Add nativeHistograms IngestionRate limit
What this PR does: As the ingestion of native histograms samples is much more CPU intensive than that of float samples - adding nativeHistogram samples specific IngestionRate limit to protect the service and to allow clients to adjust the NH series ingestion.
Checklist
- [x] Tests updated
- [x] Documentation added
- [x]
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]
Why do we need new limits for nativeHistogram? Isnt histogram already counted to the ingestionRante https://github.com/cortexproject/cortex/blob/844fa558726ca4a621b1265cc20b8f557d552da8/pkg/distributor/distributor.go#L775
Why do we need new limits for nativeHistogram? Isnt histogram already counted to the ingestionRate
NH samples are much more expensive that float samples. Float sample is always 8 bytes. A NH sample with 160 buckets and 160 spans will be ~2.5 KB in size.
NH samples are much more expensive that float samples. Float sample is always 8 bytes. A NH sample with 160 buckets and 160 spans will be ~2.5 KB in size.
I think this is more an timeseries limit, no? I am not sure the mem impact ingestionRate.
But @PaurushGarg talked to me offline. It seems there are good reason for this. Lets just update the description
There is one test failure related to this change.
--- FAIL: TestPush_QuorumError (17.76s)
distributor_test.go:878:
Error Trace: /__w/cortex/cortex/pkg/distributor/distributor_test.go:878
Error: Received unexpected error:
rpc error: code = Code(429) desc = nativeHistograms ingestion rate limit (25000) exceeded while adding 40 samples and 20 metadata
Test: TestPush_QuorumError
FAIL