Automattic
Explore ways to group notifications from CI
PR #540 improved our notification flow by only sending pings on the final failure of a test, after retry. But the Slack channel is still pretty crowded with all the other miscellaneous notes like this:
Found issue on customizer page: 'TimeoutError: Waiting for element to be located By(css selector, iframe.is-iframe-loaded)
Wait timed out after 40944ms' - Clicking try again button now. - encountered on 'chrome' at screen size 'mobile' in CircleCI build #9736 on branch 'master'
I don't want to remove those pings, as it's still useful as a general metric on how often we're hitting issues like that, but I'd like to come up with a way to contain them so the channel is easier to grok. My initial thought is to just compile all the messages for a given run into a single Snippet and upload that at the end. That should still be searchable, I think.
Light-bulb moment - let's use Slack threads - https://api.slack.com/docs/message-threading
Every build could have a master comment in the channel, and all output will be threaded underneath it. Maybe we could also keep track of that master comment and update it (https://api.slack.com/methods/chat.update) with the test status after it's done.
This could also be useful for managing builds on PRs before they're merged, as we could ping the original developer in the thread and they'd have all of their output separated from the rest, but still publicly visible/searchable.