prometheus #3351: Query labels and labels values when configured as Remote Read

Hello,

This PR is based on another PR, which is abandoned: #7076. My version is based on latest master. I have tried using client_golang for Prometheus API requests, but it does not support setting HTTP Headers, so it is unusable e.g. with Grafana Mimir with multitenancy enabled. Please review this PR. I'd be happy to apply the comments if any. Fixes #3351.

Kind regards, Szymon

Jan 23 '23 14:01 sgrzemski

@bwplotka @tomwilkie @csmarchbanks @cstyan Is there a change you can take a look at this PR? 🙏

Feb 16 '23 19:02 sgrzemski

@bwplotka thanks! Applied nits and reverted .gitignore changes.

Feb 17 '23 11:02 sgrzemski

@tomwilkie @cstyan @csmarchbanks is there a chance you can take a look here? :)

Feb 21 '23 20:02 sgrzemski

@tomwilkie @cstyan @csmarchbanks could you PTAL?

Mar 13 '23 18:03 d-mankowski-synerise

We are also missing this improvement a lot,

@tomwilkie @csmarchbanks @cstyan Is there a chance you can take a look at this PR? 🙏

May 18 '23 07:05 davidkostal

cc @bwplotka can you please follow up on this pull request?

Thanks

Jun 27 '23 11:06 roidelapluie

I prefer the original Prometheus web UI over tools like Grafana's Explorer because the way it displays multiple charts significantly improves usability. Oh pleeeease

Sep 06 '23 11:09 nodakai

@sgrzemski @bwplotka any news on this PR review? I think this will be a really nice feature to have.

Oct 18 '23 14:10 paulojmdias

Is this any closer to being merged?

I had a workaround for populating labels in Grafana but that is no longer working. This PR looks to solve the problem properly and it would be great to see the effort make it to a Prometheus release.

Nov 09 '23 17:11 ocfmatt

We have looked at this pull request during our bug scrub.

I think this should be more opt-in and we should not try to guess the URL's, but this looks interesting.

Thank you for your contribution.

Dec 05 '23 11:12 roidelapluie

Yup, explicit field in config might be more reasonable. But otherwise, interesting extension. It makes Prometheus even more fanout capable, like Thanos Querier is.

Dec 05 '23 11:12 bwplotka

Also client_golang supports headers, but with extra tripperware

Feb 21 '24 10:02 bwplotka

Do we have idea when this can be merged?

Feb 23 '24 02:02 allenmchan

I attempted to crudely pull this into a master checkout and build it but the build fails at the following output.

# github.com/prometheus/prometheus/storage/remote
storage/remote/client.go:119:60: undefined: storage.Warnings
storage/remote/client.go:120:68: undefined: storage.Warnings
storage/remote/client.go:363:39: c.url undefined (type *Client has no field or method url)
storage/remote/client.go:368:82: undefined: storage.Warnings
storage/remote/client.go:397:81: c.url undefined (type *Client has no field or method url)
storage/remote/client.go:409:23: undefined: storage.Warnings
storage/remote/client.go:418:69: undefined: storage.Warnings
storage/remote/client.go:424:84: undefined: storage.Warnings
storage/remote/read.go:212:92: undefined: storage.Warnings
storage/remote/read.go:218:78: undefined: storage.Warnings
storage/remote/client.go:409:23: too many errors
!! command failed: build -o /opt/prometheus/prometheus -ldflags -X github.com/prometheus/common/version.Version=2.52.0-rc.1 -X github.com/prometheus/common/version.Revision=3b8b57700c469c7cde84e1d8f9d383cb8fe11ab0 -X github.com/prometheus/common/version.Branch=main -X github.com/prometheus/common/[email protected] -X github.com/prometheus/common/version.BuildDate=20240514-00:19:05  -extldflags '-static' -tags netgo,builtinassets,stringlabels github.com/prometheus/prometheus/cmd/prometheus: exit status 1
make: *** [Makefile.common:204: common-build] Error 1

Have I missed something obvious or has the codebase moved on since this PR was originally opened and the pull is no longer valid?

Regards, Matt.

May 14 '24 00:05 ocfmatt

Hello from the bug scrub: we feel that the remote read API should be expanded with the required function and this implementation should be an opt-in fallback , explicitly configured.

Furthermore to expand on https://github.com/prometheus/prometheus/pull/11877#issuecomment-1840638598 , we cannot assume that the base url of remote read is the same as the other HTTP APIs, so that needs configuration.

Especially since there is now UTF-8 support, we cannot use simple Sprintf to assemble URLs from unquated strings.

Sep 24 '24 11:09 krajorama

Hello from the bug-scrub! @sgrzemski do you think you will come back to this?

Feb 25 '25 11:02 bboreham