Any predefined filters to avoid metric duplication on GKE (and overcost from Stackdriver)?

[sneko]

Hi,

Just got a bill of $600 for 2,5GB of metrics on Stackdriver within a month... surprise! haha

After investigating it seems to come from external metrics, which are pushed by Prometheus. Since Google makes it free for all metrics from nodes and pods (since I'm GKE), and that I see my own Prometheus exports also those metrics, how could I filter all metrics Google is already managing?

Does anyone of you have a list for that?

Thank you,

[AgrimPrasad]

Yes, it's mentioned in the docs. Basically, you need to include only application metrics with the prometheus sidecar and exclude everything else. https://cloud.google.com/monitoring/kubernetes-engine/prometheus#managing-costs

For example, to import only those metrics generated by kubernetes-pods and kubernetes-service-endpoints, add the following --include statement when starting the stackdriver-prometheus-sidecar:

 --include={job=~"kubernetes-pods|kubernetes-service-endpoints"}