[BUG] Alerts per bucket duplicated and notification frequency ignored, when too many buckets

[thmsbp]

What is the bug? Alerts per bucket duplicated and notification frequency broken when too many buckets

Actually, we observe 2 issues with Alert per buckets:

• Monitor Action throttling is ignored when more than 50 buckets
• Alerts (and actions and notifications) are duplicated when more than 500 buckets

How can one reproduce the bug? Steps to reproduce the behavior:

1. Define a "per bucket monitor" run every 1min
2. The monitor should observe items generating more than 500 buckets
3. Define a trigger action notification with "enable action throttling" ticked and value set to 60min

What is the expected behavior?

• Same alert notification every 60min (not every 1min)
• No duplicated alert notification

What is your host/environment?

• tested on Opensearch 2.8 and 2.12

Do you have any screenshots? no

Do you have any additional context? no

If the bug is confirmed, can you suggest any configuration to solve or workaround this behavior? This issue is criticall for us (email spamming or integration issues with third-party systems)

[thmsbp]

My issue is the same as #6710 https://github.com/opensearch-project/OpenSearch-Dashboards/issues/6710

[thmsbp]

Any opinion, any suggestion? Receiving too many alert notifications fills the mailboxes and end-users complain.

[dblock]

Doesn't look like anyone had a chance to look at it @thmsbp, sorry :( Maybe try asking for help on the public Slack?

Catch All Triage - 1 2 3 4 5 6

[thmsbp]

Thanks @dblock . This is very impactful in our system. Maybe we're the only ones experiencing these issues as a side effect of a setup or performance issues. If anyone would like to check specific logs from us, please ask.