cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

sqlstats: do not trigger flush when experiencing memory pressure

Open xinhaoz opened this issue 1 year ago • 2 comments

When writing to in-memory sql stats, we would previously trigger a flush to the system tables if we start experiencing memory pressure. In practice this resulted in the flush operation running perpetually in applications where workloads consistently hit the in-memory limits. This is problematic for a couple of reasons:

  • We don't want to this operation to run continously in the bg as constantly writing to the system tables takes up db resources.
  • Allowing adhoc flushing makes the rate of data written to the tables unpredictable per workload, unless a minimum flush interval is set. Currently, setting a minimum flush interval while experiencing memory pressure also leads to undesirable behaviour, where the sql stats flush worker still needs to process each signal which it is now rapidly receieving.

We are making the decision to remove this memory pressure signalling altogether. This simplifies the collection process and caps the rate of data being written. The tradeoff for achieving this more stable collection behaviour is to simply drop new stats when experiencing memory pressure.

If users want to retain more data they may adjust the cluster settings controlling the flush interval and capacities of in-memory containers, accepting the tradeoffs that come with flushing more frequently.

Finally, the persistedsqlstats.ApplicationStats struct is removed, since its purpose was to provide the memory pressure channel to the underlying sql stats container.

Fixes: #122161

Release note: None

xinhaoz avatar May 23 '24 23:05 xinhaoz

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] avatar May 23 '24 23:05 blathers-crl[bot]

This change is Reviewable

cockroach-teamcity avatar May 23 '24 23:05 cockroach-teamcity

TFTR! bors r+

xinhaoz avatar May 28 '24 16:05 xinhaoz