Alarm on single non-503 5xx error

[chris13524]

For chains where it is marked as guaranteed availability in SUPPORTED_CHAINS.md we should never return a 5xx error. So we should be alarmed on all 5xx errors for guaranteed chains.

• [ ] Adjust ELB 5xx alarm to ignore 503 errors because we cannot differentiate between chains. And the ELB alarm is better than a Prometheus alarm on catching more potential 5xx errors.
• [ ] Lower ELB 5xx threshold to 1
• [ ] Add a new Prometheus metric for 503 errors and make a separate alarm that alarms on a single 503 error for a guaranteed chain.

Slack conversation

[linear[bot]]

CR-17 Alarm on single 5xx error that is under SLA