Bryan Boreham comments

Results 1068 comments of


                                            Bryan Boreham

the ring never removes old ingester even if the ingester pod is evicted

"killed because of OOM" is not the same thing as "evicted". A pod that is OOM-killed will restart with the same identity on the same node, hence pick up the...

the ring never removes old ingester even if the ingester pod is evicted

Don't read too much into the words - that's removing them from one data structure in memory. There is no code to remove ingesters from the ring when they are...

the ring never removes old ingester even if the ingester pod is evicted

> what was the idea behind not removing ingesters from the ring when they are suspected to be dead? Risky, easy to get wrong, not necessary day one. > and...

the ring never removes old ingester even if the ingester pod is evicted

Here's an example scenario we want to avoid: Cortex is running under Kubernetes, and a rolling update begins: - Kubernetes sends SIGTERM to one old ingester and starts one new...

the ring never removes old ingester even if the ingester pod is evicted

I think you would find losing half the data more annoying than having to operate the system manually when there is a fault.

the ring never removes old ingester even if the ingester pod is evicted

If you indeed hit the same issue please follow the steps in https://github.com/cortexproject/cortex/issues/1521#issuecomment-513101297 If your issue is different please file it separately.

the ring never removes old ingester even if the ingester pod is evicted

The container could have restarted.

the ring never removes old ingester even if the ingester pod is evicted

Now that chunks storage is deprecated and we use blocks storage, we no longer "hand-over" from one ingester to another. So one justification for this behaviour has disappeared. Happy to...

add curl to support container probes with mTLS

I prefer not to do this; see https://github.com/cortexproject/cortex/issues/4427#issuecomment-972707651

[ruler] Allow Federated rules

Note we have always described Cortex' ability to handle separate data for each tenant as "multitenant"; we used the word "federated" when talking about querying more than one at a...