cortex icon indicating copy to clipboard operation
cortex copied to clipboard

add log to identify request cache hit and cache miss

Open mengmengy opened this issue 3 years ago • 5 comments

Signed-off-by: Mengmeng Yang [email protected]

What this PR does: Add few more logs to cache results file so that it's easier to figure out a request is skipped cache, cache hit, cache miss when a request split to multiple ones.

Checklist

  • [ ] Tests updated
  • [ ] Documentation added
  • [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

mengmengy avatar Jul 25 '22 05:07 mengmengy

We dont have metrics for it already?

alanprot avatar Jul 26 '22 21:07 alanprot

We dont have metrics for it already?

I think it logs few counters like cortex_cache_hits, cortex_cache_fetched_keys (like below), but from these, we cannot tell which split request got cache hit or cache miss. Please advise if I missed anything.

// TYPE cortex_cache_fetched_keys counter cortex_cache_fetched_keys{name="frontend.redis"} 3

// TYPE cortex_cache_hits counter cortex_cache_hits{name="frontend.redis"} 1

mengmengy avatar Jul 27 '22 22:07 mengmengy

I am not sure if we should do info logging on the call path that may be high TPS. May have impact on latency.

alvinlin123 avatar Jul 29 '22 01:07 alvinlin123

I am not sure if we should do info logging on the call path that may be high TPS. May have impact on latency.

Understood, not all logs will be executed for each split request, for each split request, only 1 log is added. Add such log will be useful when we try to figure out issues like why request is being slow etc. Or do you suggest other better ways we can figure out this I'm not aware?

mengmengy avatar Aug 03 '22 22:08 mengmengy

Maybe have it to be debug log instead of info, or maybe like @alanprot suggested, use metrics instead of logs.

alvinlin123 avatar Aug 09 '22 01:08 alvinlin123

Maybe have it to be debug log instead of info, or maybe like @alanprot suggested, use metrics instead of logs.

Metrics may not suitable for this case since it will be good to know which split request get cache hit/miss... I updated log level from info to debug.

mengmengy avatar Aug 18 '22 20:08 mengmengy

@alanprot @alvinlin123 could you please take a review about the updated PR? Thanks

mengmengy avatar Aug 23 '22 16:08 mengmengy