bookingcom
carbonzipper: introduce deterministicBackend option
What issue is this change attempting to solve?
In graphite storage (go-carbon) installations where consistency can not be guaranteed, by always prefer to return metrics from the same set of backends would make the query result less confusing, no more flapping results.
This commit essentially make graphite render result more consistent, both and right and wrong.
How does this change solve the problem? Why is this the best approach?
DeterministicBackend sets a backend weight for each metrics in render request and carbonzipper would favor metrics from a specific node. As storage backends like go-carbon does not have strong consistency guarantee, this means that the same metric persisted in different replicas might have different values. Without DeterministicBackend option, carbonzipper would just favor metrics returned first from any backend. This might result in confusing query result in carbonapi that are reading multiple metrics in one query.
An example like bellow:
asPercent(
integral(sumSeries(http.app.latencies.{0_5_1,1_5,5_10,10_25,25_50,50_100,100_250}.count)),
integral(sumSeries(http.app.latencies.*.count))
)
Without DeterministicBackend option, the example query above might return result 100.1%. With DeterministicBackend option, it is guaranteed that the query result would always "seems" meaningful, but not necessarily correct.
How can we be sure this works as expected?
Check TestMergeMetricsDifferingStepTimes7. Also confirm with production data that returning results are always consistent now.
@bom-d-van If I understand everything correctly, I am not sure this adds value to the system. Consistently returning a wrong result could be considered worse than inconsistently returning results. Inconsistency still is an indicator to the user - even if awkward - that something is not right.
Surely, it is better to let the user know that the data is wrong. This change hides the errors the system makes. It makes bad data look good.
@grzkv just read the comments here please:
https://github.com/bookingcom/carbonapi/pull/340#discussion_r802630228