HDDS-10407. Introduce Metric for deleteKey operation in SCM Service.

[muskan1012]

What changes were proposed in this pull request?

In this PR few performance metrics are introduced in SCM Service for deleteKey operation.

Please describe your PR in detail: In this PR following metrics are introduced in SCM service for deleteKey operation:

• deleteKeyFailure
• deleteKeySuccess
• deleteKeyFailureLatencyNs
• deleteKeySuccessLatencyNs

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10407

How was this patch tested?

Tested manually on the cluster.

[errose28]

Thanks for adding these metrics @muskan1012. Are we planning to add metrics for time taken by SCM's block deleting service? The metrics added here just track the time taken for SCM to add the blocks to it's list of pending deletes in the DB. The existing ScmBlockDeletingServiceMetrics class does not have any timing of the iterations.

[muskan1012]

@errose28 thank you for looking into it, yes we can add those metrics also if it will help us to track performance of deleteKey operation in SCM service. @kerneltime @duongkame do we need that metric as well?

[kerneltime]

@tanvipenumudy can you please review this PR?

[tanvipenumudy]

Tested the updated patch on a local docker-compose cluster with the monitoring add-ons using sample key deletes.

You can split the PR if you choose to address @errose28's review as part of a separate PR @muskan1012.

[muskan1012]

Sure @tanvipenumudy, we can bring it up in another pr. Thank You @tanvipenumudy for reviewing it.

[tanvipenumudy]

Thank you @muskan1012 for working on the patch, thank you @errose28 for the review.