The revision_request_count metric does not appear to resemble a counter metric.

[yabea]

Ask your question here:

Hi, I'm using the metrics of knative serving, and I see that the type of the revision_request_count metric is counter. I have also confirmed in code that it is a counter type. but the collected data is not continuosly increasing.

What could be the reason for this?

txs.

[skonto]

Hi @yabea here are some more details and example similar to the above:

• You have multiple series, one per pod, thus the different countings. For the cardinality issue check: https://github.com/knative/serving/issues/11248
• When Prometheus scrapes a pod it will get the value of counter exported at scraping time.

[shauryagoel]

Hi, I have also encountered the same issue. If no requests come for ~10 minutes, the graph starts showing no value in my case as well. Whenever a request comes, the count starts from 1. Upon digging further I found out that other metrics are also null in the same time period when there is no data in the graph. Metrics such as revision_app_request_count, revision_app_request_latencies_bucket, revision_request_latencies_bucket, revision_request_latencies_sum etc. are all null. The only metrics available are of golang- revision_go_* metrics. Whenever a request comes in I can see all of the metrics start from scratch. I checked this by entering the container and sending a curl request to http://localhost:9091/metrics.

[skonto]

@shauryagoel hi, could you add more details like screenshot of your prom query and pod status eg. does it scale down or not?

[shauryagoel]

I am using knative v1.7 and here is a sample yaml which gives the error-

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
  namespace: testing
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '9091'
    prometheus.io/path: /metrics
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/target: '10'
        autoscaling.knative.dev/min-scale: '1'
        autoscaling.knative.dev/max-scale: '100'
    spec:
      containers:
        - image: ghcr.io/knative/helloworld-go:latest
          ports:
            - containerPort: 8080
          env:
            - name: TARGET
              value: World

The pod does not scale down as min scale is set to 1 above. Still I am getting the same error. Prom query- revision_app_request_count{configuration_name="hello"} Output is similar to what is attached in the issue; data is missing.

[skonto]

Could you try with a community supported version? 1.7 is too old.

[shauryagoel]

Hi, is there some parameter which adjusts the duration of the metrics export? This parameter is most likely set to around 10min as discussed previously. I am currently unable to deploy to a higher version of knative as we have a knative dependency of <=1.9.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.