Bartlomiej Plotka
Bartlomiej Plotka
I don't have time to take a deeper look, but generally main looks better than this PR on prombench, unfortunately. Not significantly, but there is some diff: CPU is 6%...
All metrics with scrape_job has the same issue e.g. `prometheus_target_sync_length_seconds` but they might have some value for some advanced debugging cases. https://github.com/prometheus/prometheus/blob/bfbb13cf369da4cd1b29ee52c396c902723febfb/scrape/metrics.go#L138
Retro link: http://prombench.prometheus.io/grafana/d/7gmLoNDmz/prombench?orgId=1&from=1713875299363&to=1714140211760&var-RuleGroup=All&var-prNumber=13969
Also noticed uneven traffic to services, explaining latency being higher for Prom from this PR for the period where it has more request to serve than the old one. Unfortunately...
Unfortunately we see even 0.9 percentile difference for moments the load was even. It seems indeed we have a small regression in query tail latency in this release, but perhaps...
For 0.99 it was sometimes extreme difference - correlating with CPU does not help (maybe a bit more CPU used, weird) 
I think that one was affected by compaction (unlucky) 
I think I would be supportive here, but I would love to see the full PR with minimal changes here. (Just speaking with @qinxx108 on Prometheus booth 💪🏽 ) Something...