"Application metrics" not being exported after setting ENABLE_METRICS

[davidpellcb]

Chart version: 3.6.2

We've set ENABLE_METRICS:

/ $ strings /proc/1/environ | grep -i metrics
ENABLE_METRICS=true

...however, the metrics chartmuseum_charts_served_total and chartmuseum_chart_versions_served_total documented here aren't being exported.

❯ curl -s localhost:8080/metrics | grep -v "#" | grep chartmuseum_chart | wc -l
       0

The "general global metrics" are being exported:

❯ curl -s localhost:8080/metrics | grep -v "#" | grep -i chartmuseum
chartmuseum_request_duration_seconds_sum 11.347815388000111
chartmuseum_request_duration_seconds_count 330745
chartmuseum_request_size_bytes_sum 2.8782004e+07
chartmuseum_request_size_bytes_count 330745
chartmuseum_requests_total{code="200",handler="helm.sh/chartmuseum/pkg/chartmuseum/router.(*Router).rootHandler-fm",host="x:8080",method="GET",url="/health"} 330192
chartmuseum_requests_total{code="200",handler="helm.sh/chartmuseum/pkg/chartmuseum/router.(*Router).rootHandler-fm",host="x:8080",method="GET",url="/:repoindex.yaml"} 553
chartmuseum_response_size_bytes_sum 4.34963519e+08
chartmuseum_response_size_bytes_count 330745

[cbuto]

@davidpellcb do you see any logs that might help identify the issue? or maybe share your full configuration (redacted) so we can try to reproduce?

I ran ChartMuseum v0.14.0 locally and saw the metrics being exported.

[mzupan]

@cbuto did you test against an object store like s3? it works fine if i have local storage but fails for s3

[mzupan]

ok metrics work fine in my testing on s3.. the issue is if app dies there no cache or state. If you do another push for a chart the metrics come back.

[jlm0x017]

Did some poking at this. It's not just "chart/version served total" metrics that are delayed. "Requests total" is delayed as well; not showing in /metrics until after the first call is made.

$ curl -s http://localhost:8080/metrics | grep -v '^#' | grep -E '^chartmuseum'
chartmuseum_request_duration_seconds_sum 0
chartmuseum_request_duration_seconds_count 0
chartmuseum_request_size_bytes_sum 0
chartmuseum_request_size_bytes_count 0
chartmuseum_response_size_bytes_sum 0
chartmuseum_response_size_bytes_count 0
$ curl -s http://localhost:8080/metrics | grep -v '^#' | grep -E '^chartmuseum.*total'
$ curl  http://localhost:8080/api/charts | jq '.' | head -5
{
  "redacted": [
    {
      "name": "redacted",
      "version": "0.2.2",
$ curl -s http://localhost:8080/metrics | grep -v '^#' | grep -E '^chartmuseum.*total'
chartmuseum_requests_total{code="200",handler="helm.sh/chartmuseum/pkg/chartmuseum/router.(*Router).rootHandler-fm",host="localhost:8080",method="GET",url="/:repoapi/charts"} 1

I'm suspicious that this upstream bug is related: https://github.com/zsais/go-gin-prometheus/issues/55

I was hoping that forcing a cache or index refresh would trigger the stats to appear; I ran chartmuseum alternatively with --cache-interval=30s and --always-regenerate-chart-index but neither was getting the chart/version totals to appear.

Finally, I'll add, @mzupan already notes a POST will cause the metrics to start appearing. So will a DELETE. This is perhaps easier to trigger:

$ curl -X "DELETE" http://localhost:8080/api/charts/nonexistant/0.1.0
{"deleted":true}

testing was done with the v0.16.0 docker container on my local machine, but pointed at our s3 bucket.