PagerDuty and structured alerts data

[sylr]

I feel there is a need for a new PagerDuty integration in the Alertmanager which would keep alerts data in a structured form when transmitted to PagerDuty as opposed to the currently templated form.

I believe we need this because PagerDuty offers Automation features which allow to take certain actions based on field values. The fact that Alertmanager marshalls annotations and labels values in big chunks of text in the firing or resolved custom details fields limit greatly the possibility of using these data in automation rules.

PagerDuty V2 Event API support has a custom_details field which is currently used like this:

"custom_details": {
  "firing":       `{{ template "pagerduty.default.instances" .Alerts.Firing }}`,
  "resolved":     `{{ template "pagerduty.default.instances" .Alerts.Resolved }}`,
  "num_firing":   `{{ .Alerts.Firing | len }}`,
  "num_resolved": `{{ .Alerts.Resolved | len }}`
}

Unless I'm mistaken, the keys of the custom_details object must be hardcoded in the receiver configuration. There is no possibility to define them dynamically using .Alerts.(Resolved|Firing).Labels or .Alerts.(Resolved|Firing).Annotations.

After some testing it seems that PagerDuty V2 Event API supports at least 1 level of objects in custom_details allowing to do something like that.

PSEUDO CODE that describes valid JSON map[string]map[string]string for custom_details:

"custom_details": {
  "annotations": {
{{ range .Alerts.(Resolved|Firing)[0].Annotations.SortedPairs }}
    "{{ .Name }}": "{{ .Value }}",
{{ end }}
  },
  "labels": {
{{ range .Alerts.(Resolved|Firing)[0].Labels.SortedPairs }}
    "{{ .Name }}": "{{ .Value }}",
{{ end }}
  },  
}

In that form, I think it will offer Alertmanager users the possibility to use advanced PagerDuty features.

Regards.

[porkcharsui]

I totally agree that Alertmanager's PagerDuty integration really needs to support structured data for outbound alerts. I also tried customizing my text/template output to capture structured data like you proposed but ended up falling back to another approach.

I configure Alertmanager to output via a generic webhook, and target this to hit a minimal webhook transformer service I wrote running adjacent to Alertmanager. This service listens on localhost and takes the inbound webhook event and creates a PagerDuty V2 Event using the client API library. This way I fully preserve the entire inbound event structure without modifying Alertmanager.

[NargiT]

I will link a related issue, a bit older, but still relevant. https://github.com/prometheus/alertmanager/issues/2477

Event nagios has a better integration that alert manager which should be more modern.