conjure-up
WIP get service and pod metrics via our prometheus scraper
This isn't quite ready yet, but it brings in the latest service and pod metrics functionality from the upstream prometheus scraper:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
Since our prometheus is not in-cluster, we have to get svc/pod metrics via the load balancer. I lifted relabeling configs to do this from the following gist:
https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450
It's still not quite right though:
As an example into the red, see what prometheus is effectively trying to do for the nginx-ingress pod:
$ curl -u admin:PASSWORD -k https://KUBEAPI_LOAD_BALANCER:443/api/v1/namespaces/default/pods/nginx-ingress-kubernetes-worker-controller-jp962:443/proxy/metrics
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.13.12</center>
</body>
</html>
You might say, "oh hey, just stop trying to hit pod:443 and you'll get past this". I'd be like, "OK", and then show you this:
$ curl -u admin:bm0fGt3hFNudiYc8GYbCIg8KsEURSME3 -k https://172.31.23.115:443/api/v1/namespaces/default/pods/nginx-ingress-kubernetes-worker-controller-jp962/proxy/metrics
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>
We do have some green, which makes me think we're on the right track. It could be some of these things are not producing metrics, or we're looking for them in the wrong place (maybe not proxied how we expect). I wanted to put this up to solicit ideas from anyone who may know how to scrape pods/services from a prometheus that is not in-cluster.
NOTE: to even get what we have now, we have to annotate pods and services so that the scraper sees them from the api server:
kubectl annotate pod --namespace='[default|kube-system]' --all prometheus.io/scrape=true
kubectl annotate svc --namespace='[default|kube-system]' --all prometheus.io/scrape=true
