panic icon indicating copy to clipboard operation
panic copied to clipboard

Chainlink Alerting: Prometheus Balance Metric Investigation

Open Cherrett opened this issue 3 years ago • 0 comments

Rationale

Recently we have released a hot-fix which caters for the following eth_balance structure found in the prometheus of Chainlink-based nodes:

eth_balance{account="0x12345",evmChainID="123"} 123.456

The evmChainID is used to get the currency moniker for the respective chain.

It was noted that on some Chainlink-based chains, this field is non-existent and is instead replaced with:

example_balance{account="0x12345",chainID="mainnet",chainSet="example",denomination="ABC"} 123.456

We must investigate the prometheus balance metric of different Chainlink-based chains/nodes to see how we should proceed.

One solution might be to search for the balance substring in all of the field names. This will then be parsed accordingly:

  • Use already existing logic if we have account and evmChainID fields.
  • Use new logic for using denomination as the currency moniker if we don't have the evmChainID field and have the account and denomination fields.

For ticket closure

  • Have a list of different structures found in prometheus sources of various Chainlink-based chains/nodes.
  • Decide on what will be done in the alerter to cater for these variations.
  • Open tickets based on the point above.

Cherrett avatar May 17 '22 13:05 Cherrett