prometheus
The Ability To Add Labels or Annotations By Route or To Webhooks
What feature would you like to see? I have a webhook integration, which expects a static label or annotation to exist in all hooks. I would like the ability to add static labels or annotations to all alerts that are sent by route.
Or, I would like the ability to add static labels or annotations to a webhook config (common_labels or common_annotations).
By design, Alertmanager doesn't (can't) manipulate the alert labels. If anything can be done, I guess that it would be at the Prometheus level with alerting_relabel_configs.
Other receiver configs can be templated to add semantically useful information to downstream alerting. Why not webhook_configs?
@midnightconman I'm not sure I follow what kind of problem this is trying to solve. Labels and annotations have to come from Prometheus, so the webhook integration has to handle missing values.
If the goal was to know which receiver sent the notification, you can add custom query params to your webhook url in the alertmanager config, or diff http configs. Can you elaborate a little bit more on why this is necessary for your use case?
Other receiver configs can be templated to add semantically useful information to downstream alerting. Why not webhook_configs?
Other fields that can be templated are content/template strings. WebhookConfig fields are just urls and http configs, and templating isn't supported for those in any other receiver either.
@waltherlee, from what I recall, which isn't much, I was trying to integrate with GitLab's Prometheus compatible API and trying to avoid needing to introduce a middle translation or handling logic. Introducing labels to the web hook configuration would mean that intermediary tools could be agnostic to the routes and let later database batch/ui filtering know what they mean.
@djmcgreal-cc Yeah, I'm actually wrong, the alertmanager already supports templating in some URL fields, for example, here.
I think adding support for webhook makes sense too. Labels and annotations should be managed at the Prometheus level, so allowing different configs based on the values in the alert aligns with that.
So I think we're good and this can be closed. Adding labels/annotations is not something the alertmanager will do, but supporting templating in webhook sounds reasonable. I'll reopen https://github.com/prometheus/alertmanager/issues/684 and close this one as duplicate.