Horizon - Identify memory consumption model (Captive Core + Horizon)

[jcx120]

Captive Core requires 22GB of RAM to run (as of April/22) for ingestion and each reingestion worker. This has increased from 12GB earlier this year (due to ledger size growth).

However, during initialization, there could be additional memory requirements beyond 22GB RAM that could spike and prevent the process from proceeding (and settling back down to 22GB in steady state).

Tasks for this issue:

• [ ] Identify minimum RAM needed: Run a test to monitor memory consumption from startup to continuous ingestion process identify the amount of RAM to account for startup spikes to steady state
• [ ] Graph the projected rate of growth per month (based on ledger growth trend) and model the expected requirements for RAM increase per month (ie: 1.2x/month or if linear, +1GB/month)

[bartekn]

Last week I run ingest.CheckpointChangeReader without anything else and it consumes 3.6GB at one of the latest checkpoints but database ingestion instances can consume even 8GB. We need to understand where 4.4GB come from.