exception_notification
exception_notification copied to clipboard
Error counts for grouped exception notifications are updated inconsistently, causing duplicate alerts
Steps to reproduce
- Use Redis as a cache to group notifications and deploy app onto Heroku using Heroku-20 Stack. (This bug does not appear during local development.)
- Trigger an exception 8 times within the default grouping period of 5 minutes. See that notifications are triggered appriopriately.
- Wait for the default grouping period of 5 minutes to pass.
- Do step 2 again. See that notifications are not triggered appropriately. The second alert is duplicated.
| Step 2 | Step 4 |
|---|---|
Expected behavior
Notifications should always be triggered appropriately and not duplicated.
Actual behavior
Sometimes, notifications are duplicated. When investigating the Redis cache, it looks like ExceptionNotifier sometimes switches from updating and using the error count from the message-based key to the error count from the backtrace-based key even though it is supposed to just hit the message-based key consistently.
For example, in step 4, alerts are triggered in the following way:
- Trigger first exception: message-based key has error count of 1 backtrace-based key has error count of 1 First alert is triggered
- Trigger second exception message-based key has error count of 2 backtrace-based key has error count of 1 Second alert is triggered with "(2 times)" in title
- Trigger third exception message-based key has error count of 3 backtrace-based key has error count of 1 No alert is triggered
- Trigger fourth exception message-based key has error count of 3 backtrace-based key has error count of 2 Third alert is triggered with "(2 times)" in title
- Trigger fifth exception message-based key has error count of 3 backtrace-based key has error count of 3 No alert is triggered
- Trigger sixth exception message-based key has error count of 4 backtrace-based key has error count of 3 Fourth alert is triggered with "(4 times)" in title
System configuration
Rails version: 6.1.7.3
Ruby version: 3.0.5
Other configurations: Sidekiq: 6.5.9 Redis: >= 4.5.0, < 5 (we get Redis from importing Sidekiq)