cortex icon indicating copy to clipboard operation
cortex copied to clipboard

Add nativeHistograms IngestionRate limit

Open PaurushGarg opened this issue 7 months ago • 4 comments

What this PR does: As the ingestion of native histograms samples is much more CPU intensive than that of float samples - adding nativeHistogram samples specific IngestionRate limit to protect the service and to allow clients to adjust the NH series ingestion.

Checklist

  • [x] Tests updated
  • [x] Documentation added
  • [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

PaurushGarg avatar Jun 05 '25 14:06 PaurushGarg

Why do we need new limits for nativeHistogram? Isnt histogram already counted to the ingestionRante https://github.com/cortexproject/cortex/blob/844fa558726ca4a621b1265cc20b8f557d552da8/pkg/distributor/distributor.go#L775

danielblando avatar Jun 05 '25 14:06 danielblando

Why do we need new limits for nativeHistogram? Isnt histogram already counted to the ingestionRate

NH samples are much more expensive that float samples. Float sample is always 8 bytes. A NH sample with 160 buckets and 160 spans will be ~2.5 KB in size.

harry671003 avatar Jun 05 '25 16:06 harry671003

NH samples are much more expensive that float samples. Float sample is always 8 bytes. A NH sample with 160 buckets and 160 spans will be ~2.5 KB in size.

I think this is more an timeseries limit, no? I am not sure the mem impact ingestionRate.

But @PaurushGarg talked to me offline. It seems there are good reason for this. Lets just update the description

danielblando avatar Jun 05 '25 17:06 danielblando

There is one test failure related to this change.

 --- FAIL: TestPush_QuorumError (17.76s)
    distributor_test.go:878: 
        	Error Trace:	/__w/cortex/cortex/pkg/distributor/distributor_test.go:878
        	Error:      	Received unexpected error:
        	            	rpc error: code = Code(429) desc = nativeHistograms ingestion rate limit (25000) exceeded while adding 40 samples and 20 metadata
        	Test:       	TestPush_QuorumError
FAIL

harry671003 avatar Jun 06 '25 05:06 harry671003