[WIP] Add Operator Observability Best Practices

[sradco]

Signed-off-by: Shirly Radco [email protected]

Description of the change: In this document we will outline what operators require in order to meet the "Deep Insights" capability level and provide best practices and examples for creating metrics, recording rules and alerts.

Motivation for the change: Standardize operators observability, in order to be able to create useful tooling for operator developers and be able to certify operators "Deep Insights" capability level based on the requirements and test them regularly using CI.

Checklist

If the pull request includes user-facing changes, extra documentation is required:

• [ ] Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
• [ ] Add or update relevant sections of the docs website in website/content/en/docs