garnet icon indicating copy to clipboard operation
garnet copied to clipboard

Enforce object store total memory size constraint during recovery

Open yrajas opened this issue 1 year ago • 0 comments

Problem Whenever object store operations are performed, corresponding heap size impact is computed and tracked to ensure that the total memory size is within the user configured limit. However, this constraint is not currently enforced during recovery. This change addresses that.

Fix description

  • Ability to register a deserialization observer, OnDeserializationObserver, that can compute and use the CacheSizeTracker to track heap usage when pages are read in during recovery. LogOperationObserver is added to perform this.
  • During recovery, if total object store size needs to be tracked, the log memory is not fully saturated by reading pages from disk rather done sequentially.
  • If reading in a page causes the memory limit to be breached, earlier pages are evicted EvictPage one at a time until the total memory size is back within the limit.
  • If pages were freed to constrain memory limits, DoPostRecovery updates headAddress to account for the freed pages.

yrajas avatar May 05 '24 16:05 yrajas