Update Cosmos expire metrics/alerts to use unique redis alert store keys

[Cherrett]

Technical Story

As the alert store, I want to avoid overwriting slashed alerts and new/concluded proposal alerts.

Description

Currently, we are storing slashed alerts in a single redis key while storing new/concluded proposal alerts in two redis keys, one for each alert. This is not ideal as multiple alerts of the same type within a given time frame will overwrite the previous alerts. To solve this issue, we must have a unique redis alert key for each slashing event (might need to use block ID) and proposal ID. This is similar to what we have for chainlink contracts. These will be set with an expiry value and will be removed by a future component (#204).

Requirements

• Update alert_cosmos_nodeX redis alert store keys and corresponding functions for slashing events to use block ID postfix
• Update alert_cosmos_networkX redis alert store keys and corresponding functions for proposals to use proposal ID postfix
• Add logic in alert store to cater for the new redis alert store keys
• Update API/UI to cater for these new redis alert store keys
• Update tests accordingly

Blocked by

N/A

Acceptance criteria

Scenario: The CosmosNodeStore is started Given: PANIC started When: Checking the log files inside the alerter/logs/stores and the docker-compose console Then: You can see correct startup

Scenario: The CosmosNetworkStore is started Given: PANIC started When: Checking the log files inside the alerter/logs/stores and the docker-compose console Then: You can see correct startup

Scenario: The CosmosNodeStore has no error logs Given: PANIC is running When: Checking the log files inside the alerter/logs/stores Then: You can see no error logs

Scenario: The CosmosNetworkStore has no error logs Given: PANIC is running When: Checking the log files inside the alerter/logs/stores Then: You can see no error logs

Scenario: The CosmosNodeStore is receiving alerts from the CosmosNodeAlerter Given: PANIC is running in Debug mode (by putting LOGGING_LEVEL=INFO inside the .env file and restarting panic) When: Checking the log files inside the alerter/logs/stores Then: You can see the correct alerts being received.

Scenario: The CosmosNetworkStore is receiving alerts from the CosmosNetworkAlerter Given: PANIC is running in Debug mode (by putting LOGGING_LEVEL=INFO inside the .env file and restarting panic) When: Checking the log files inside the alerter/logs/stores Then: You can see the correct alerts being received.

Scenario: The CosmosNodeStore is storing alerts correctly Given: PANIC is running in Debug mode (by putting LOGGING_LEVEL=INFO inside the .env file and restarting panic) When: Checking the log files inside the alerter/logs/stores Then: You can see the correct alerts stored in unique keys in Redis.

Scenario: The CosmosNetworkStore is storing alerts correctly Given: PANIC is running in Debug mode (by putting LOGGING_LEVEL=INFO inside the .env file and restarting panic) When: Checking the log files inside the alerter/logs/stores Then: You can see the correct alerts stored in unique keys in Redis.

Scenario: The Telegram Commands Handler and Slack Commands Handler do not show any issues on /status and /panicstatus commands respectively. Given: PANIC is running with a Slack Commands Handler and Telegram Commands Handler When: Typing /status and /panicstatus Then: You can see that PANIC is running smoothly