Performance Regression Checks

[jflo]

Coordinate with Ameziane and performance workstream on this https://github.com/Consensys/protocol-misc/issues/831

• All performance testing requires consistent VM specs.
• Discourage use of GHA hosted free runners, as we have no control over the quality.

For any given build, we would like to automate its deployment, and mark its time of startup as well as when it finishes syncing and begins following head with the rest of the network. All interesting metrics should be collected during this time, and should be comparable to a "baseline" set of data. If divergence from that baseline is outside of an accepted deviation, alerts should be fired.

Examples of interesting metrics:

• Time to sync
• Block processing time
• Time to acquire max peers
• CPU, Memory use and Disk IO use both during sync and “following head”.

Strawman Design:

• GithubAction or Webhook fires when a Release Candidate is created.
• Some process reacts to that and deploys that build in N permutations to Consensys managed cloud infrastructure. N would include a variety of CL clients and test networks.
• Each node would log a "startup" event via prometheus metrics
• Each node would start syncing from an empty state
• Each node would log a "synced" event via prometheus metrics
• Alerts would be developed in grafana which would fire when any of the metrics of interest diverge from norms.