Horizon: Review and adjust monitoring / alerting for Soroban data

[jcx120]

Review scope of changes from Soroban and impact to Horizon eg:

• ingestion and processing on new TX Meta events
• Deriving account balances w/ Soroban data
• Anticipated impacts to ingestion time, query performance that we want to monitor

Alerts / monitoring - tasks:

• [ ] Identify changes to alerting / monitoring to capture any new data or update existing metrics (grafana / prometheus)
  • [ ] Adjust alerts to take into changes for ie:
  • Ingestion duration / delay
  • request duration for impacted endpoints
  • state correctness
  • DB health
  • Disk warning levels etc
  • Memory
  • Anything else we deem to be risks
• [ ] Create Ops ticket with changes to Prometheus / grafana settings
• [ ] Apply changes to Staging environment

[jcx120]

cc @mollykarcher