Bryan Boreham

Hi, I can pick up this PR, rebase it to latest and apply the changes suggested by @everett980. Would you like me to do that?

See also https://github.com/grafana/jsonnet-libs/pull/762; lowering `ndots` strictly speaking breaks Kubernetes service discovery, but if you remember not to rely on the specific bit you'll be ok. [EDIT: was previously linking to...

Interesting idea. Where would this code run? Would there be a daemon beside the Grafana instance running Python code?

Yes, what the code calls “user”, but sometimes “org”. Cortex is “multi-tenant”. I’d argue that a significant percentage of users will never use multi-HA, and it’s no great problem to...

@Alphasaurs thanks for volunteering! If you take the latest code and cherry-pick Steve's commit here: https://github.com/stevesg/cortex/commit/b0a95a22cf06f441aa4e4f830df63d14defe83db then `go test pkg/distributor` should show part of the problem. Let us know how...

Broadly the pattern is that you have to find what is starting the goroutines, find out how they are supposed to be stopped, and add or fix the code in...

We could put errors like this in `WriteResponse` instead of `error`, and extend the distributor to understand such responses and send back an error to the caller without logging it.

There is some work in https://github.com/weaveworks/common/pull/195 to rate-limit "high volume errors", ~however as it stands it would not affect 400 errors.~ (not directly relevant as it's for http and these...

To update some of the earlier analysis since code may have moved: The error logged by the distributor comes from here: https://github.com/cortexproject/cortex/blob/8587ea61fe17d71da529917c9e48b548f4dbe8ef/pkg/util/push/push.go#L47 and looks like this: ``` level=error ts=2020-09-24T04:06:45.73264618Z caller=push.go:47...

@dmares01 by all means take a look. We don't think anyone is actively looking at this issue.