Discrepancy between the QPS reported by activator and queue-proxy when activator is in the request path

[lichenran1234]

In what area(s)?

/area monitoring

Ask your question here:

We are using knative 0.23.2. We set target-burst-capacity = -1 to make sure the activator is always in the request path. And contianerConcurrency is not set. We were load testing a revision with only 1 replica (min/max replicas set both to 1). We expect all requests to go through activator, but we got different QPS by looking at the metrics activator_request_count, revision_app_request_count and revision_request_count:

• revision_app_request_count (QPS in user container) and revision_request_count (QPS in queue proxy) consistently reported the same QPS as seen by our load testing client.
• activator_request_count (QPS in activator) consistently reported less QPS than the two above (QPS seen by the activator was 50% ~ 70% of the QPS seen by queue proxy and user container).

Question: when the activator is in the request path for a revision, does 100% of the revision's traffic go through it? If so, why would we see discrepancy in the different request_count metrics?

[nader-ziada]

Can you also check the activator capacity?

https://knative.dev/docs/serving/load-balancing/activator-capacity/#setting-the-activator-capacity

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.