influxdata
Support Prometheus Operator
Feature Request
Currently, Telegraf supports the Prometheus annotations for service discovery in Kubernetes. With CRDs hitting GA, there's growing support for this workflow within the Kubernetes community; particularly around Prometheus.
The Prometheus Operator provides CRDs for directing Prometheus servers to metric endpoints, which Telegraf should support too.
Proposal:
Support the ServiceMonitor CRD, with the same behaviour as the annotation approach.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-app
labels:
team: frontend
spec:
selector:
matchLabels:
app: example-app
endpoints:
- port: web
Current behavior:
Not supported
Desired behavior:
Scrape metrics as per instructions in ServiceMonitor CRD
Use case:
The Prometheus Operator is becoming the standard for Kubernetes monitoring and Telegraf could provide a drop-in replacement
High level sounds good, I'm assuming you are referring to the CoreOS Prometheus Operator? Can you pseudocode the calls we would make and what example responses would look like?
@danielnelson, you need to support the definition of the destination port, getting them not from labels, but from spec.ports[%PORT_NAME%]
Example service manifest:
apiVersion: v1
kind: Service
metadata:
name: pili
labels:
app: pili
spec:
type: ClusterIP
ports:
- name: uwsgi
protocol: TCP
port: 8080
targetPort: uwsgi
selector:
app: pili-web
tier: backend
Then the configuration of Prometheus is something like this:
[[inputs.prometheus]]
...
kubernetes_lablel_selector = "appl=pili"
kubernetes_monitor_endpoint = "uwsgi"
kubernetes_monitor_path = "/metrics"
kubernetes_monitor_scheme = "http"
...
In this form, it will be necessary to create one input line per service, but this is seen as a plus rather than a minus. It might be worth using an array of endpoints:
[[inputs.prometheus]]
[[inputs.prometheus.endpoints]]
selector = "appl=pili"
endpoint = "uwsgi"
path = "/metrics"
scheme = "http"
...
But, as it seems to me, such a refinement will cost more.
ServiceMonitor configuration example for the above service:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: monitoring-pili
namespace: monitoring
labels:
app: pili-service-monitor
spec:
selector:
matchLabels:
# Target app service
app: pili
endpoints:
- interval: 15s
path: /metrics
port: uwsgi
scheme: http
namespaceSelector:
matchNames:
- default
This logic allows you to use the same principle of collecting goals as ServiceOperator.
Or you can use the API of the prometheus operator - https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#servicemonitor
But this will require a deployed prometheus operator in the cluster, which, in some scenarios, I would like to avoid.
@M0rdecay work on this support actually started last week! We'll keep you updated with this issue 👍
