Added queryIndex cleanup job

[petardz]

Issue #, if available: #826

Description of changes:

Added a job which is periodically checking if backing query index can be deleted. Deletion of index is done in following conditions:

1. Backing query index IS NOT writeIndex
2. All percolator query documents are related to source indices which does not exist anymore

CheckList: [ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #830 (a3f687c) into main (89142f4) will decrease coverage by 0.32%. The diff coverage is 82.42%.

:exclamation: Current head a3f687c differs from pull request most recent head 7e30cc4. Consider uploading reports for the commit 7e30cc4 to get more accurate results

@@             Coverage Diff              @@
##               main     #830      +/-   ##
============================================
- Coverage     76.07%   75.76%   -0.32%     
- Complexity      110      111       +1     
============================================
  Files           125      126       +1     
  Lines          7065     7015      -50     
  Branches       1052     1052              
============================================
- Hits           5375     5315      -60     
- Misses         1153     1157       +4     
- Partials        537      543       +6     

Impacted Files	Coverage Δ
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt	76.56% <60.00%> (-1.41%)	:arrow_down:
...n/kotlin/org/opensearch/alerting/AlertingPlugin.kt	93.25% <66.66%> (-0.50%)	:arrow_down:
...alerting/transport/TransportDeleteMonitorAction.kt	75.53% <78.94%> (+9.23%)	:arrow_up:
...g/opensearch/alerting/util/QueryIndexManagement.kt	83.69% <83.69%> (ø)
.../org/opensearch/alerting/MonitorMetadataService.kt	69.73% <86.36%> (+4.35%)	:arrow_up:
...g/opensearch/alerting/settings/AlertingSettings.kt	98.94% <100.00%> (+0.04%)	:arrow_up:
...opensearch/alerting/util/DocLevelMonitorQueries.kt	72.59% <100.00%> (-0.49%)	:arrow_down:

... and 16 files with indirect coverage changes

[dblock]

What shall we do with this @petardz? Close? Finish?

[Catch All Triage - 1, 2, 3, 4]

[petardz]

@dblock this one was covering some cases that previous PR missed.

I think we had brief discussion long time ago, to simplify this a bit by adding metadata in mappings to every queryIndex so that "queryIndex cleanup job" can have generic way of listing all queryIndexes on cluster.