observability icon indicating copy to clipboard operation
observability copied to clipboard

Per-node (and per-shard?) CPU utilization should be a heatmap

Open hcoyote opened this issue 2 years ago • 1 comments

Current design is an overall cluster utilization up to 100%. This masks two things:

  1. Makes it hard to see if one broker is heavily over-utilized/under-utilized
  2. Makes it hard to understand actual cpu utilization if you don't know the underlying core counts configured for shards.

It would be good to break up the primary cpu utilization gauge into at least a per-broker heat map (and maybe a per-shard one on one of the lesser dashboards, like the ops dashboard, rather than the default).

I think this will help people understand available headroom given that they're likely used to looking at system.cpu from the kernel where this stuff is more easily accessible, especially if coming from another system like Apache Kafka where java utilization == cpu utilization.

hcoyote avatar Jun 13 '23 21:06 hcoyote

This should also make sure to use deriv() instead of rate() since redpanda_cpu_busy_seconds is (accidentally) a guage instead of acounter.

hcoyote avatar Oct 23 '24 20:10 hcoyote