prometheus-community
Add option for clients to forward requests from the host IP to 127.0.0.1
If some services listens to localhost instead of the node's FQDN/public ip, it's currently not possible for the client to scrape their metrics.
This feature is already implements in the Rancher's PushProx fork
Maybe it is not a problem anymore but I can't reproduce your use case.
Can you specify the config please ?
Here is a try:
start proxy (listen on :8080)
$ ./pushprox_proxy &
ts=2024-06-01T15:22:35.877Z caller=main.go:373 level=info msg=Listening address=:8080
start client identified by localhost and connected to proxy
$./pushprox-client --fqdn=localhost --proxy-url=http://192.168.0.196:8080 &
ts=2024-06-01T15:25:07.579Z caller=main.go:243 level=info msg="URL and FQDN info" proxy_url=http://192.168.0.196:8080/ fqdn=localhost
try to scrap an exporter (httpapi_export bind on localhost
curl --proxy 127.0.0.1:8080 -H "X-Prometheus-Scrape-Timeout-Seconds: 500" http://localhost:9321
<html>
<head>
<title>Prometheus httpapi_exporter</title>
<style type="text/css">
body { margin: 0; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 1.42857143; color: #333; background-color: #fff; }
.navbar { display: flex; background-color: #222; margin: 0; border-width: 0 0 1px; border-style: solid; border-color: #080808; }
.navbar > * { margin: 0; padding: 15px; }
.navbar * { line-height: 20px; color: #9d9d9d; }
.navbar a { text-decoration: none; }
.navbar a:hover, .navbar a:focus { color: #fff; }
.navbar-header { font-size: 18px; }
body > * { margin: 15px; padding: 0; }
pre { padding: 10px; font-size: 13px; background-color: #f5f5f5; border: 1px solid #ccc; }
h1, h2 { font-weight: 500; }
a { color: #337ab7; }
a:hover, a:focus { color: #23527c; }
table { border: 1px solid #edd2e6; border-collapse: collapse; margin-bottom: 1rem; width: 80%; }
tr { border: 1px solid #edd2e6; padding: 0.3rem; text-align: left; width: 35%; }
th { border: 1px solid #edd2e6; padding: 0.3rem; }
td { border: 1px solid #edd2e6; padding: 0.3rem; }
.odd { background-color: rgba(0,0,0,.05); }
</style>
</head>
<body>
<div class="navbar">
<div class="navbar-header"><a href="/">Prometheus httpapi_exporter</a></div>
<div><a href="/healthz">Health</a></div>
<div><a href="/metrics">Metrics</a></div>
<div><a href="/config">Configuration</a></div>
<div><a href="/targets">Targets</a></div>
<div><a href="/status">Status</a></div>
<div><a href="/debug/pprof">Profiling</a></div>
<div><a href="/httpapi_exporter_metrics">Exporter Metrics</a></div>
<div><a href="https://github.com/peekjef72/httpapi_exporter#readme">Help</a></div>
</div>
<p>This is a <a href="https://github.com/peekjef72/httpapi_exporter#readme">Prometheus httpapi_exporter</a> instance.
You are probably looking for its <a href="/metrics">metrics</a> handler.</p>
</body>
</html>
and logs form both proxy and client
ts=2024-06-01T15:38:45.694Z caller=coordinator.go:122 level=info msg=DoScrape scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 url=http://localhost:9321/
ts=2024-06-01T15:38:45.694Z caller=main.go:208 level=info msg="Responded to /poll" url=http://localhost:9321/ scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36
ts=2024-06-01T15:38:45.695Z caller=main.go:206 level=info msg="Got scrape request" scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 url=http://localhost:9321/
ts=2024-06-01T15:38:45.695Z caller=coordinator.go:143 level=info msg=WaitForScrapeInstruction fqdn=localhost
ts=2024-06-01T15:38:45.696Z caller=main.go:140 level=info scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 msg="Retrieved scrape response"
ts=2024-06-01T15:38:45.697Z caller=main.go:189 level=info msg="Got /push" scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36
ts=2024-06-01T15:38:45.697Z caller=coordinator.go:175 level=info msg=ScrapeResult scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36
ts=2024-06-01T15:38:45.697Z caller=main.go:146 level=info scrape_id=ca00a77d-4cc2-429b-95fc-28e4f17d4d36 msg="Pushed scrape result"
Thanks a lot @peekjef72 :) I currently use the pushprox-client from the Rancher project, but I'm considering switching to the upstream version. I'll be testing it out in the coming days.
@peekjef72 If there are multiple replicas listening on the same local port of different machines, how to distinguish them from prometheus side since all of the fqdn should be "localhost"
