sensay-nelson

Looking closer, I'm seeing a lot of these errors in the kube-state-metrics pods So perhaps my issue is permissions related. Do we know what permissions this container requires? I do...

I fixed the permission issues by adding a serviceaccount to the kubernetes configuration for kube-state-metrics with the following configs. Unfortunately, it did not improve any information available in the dashboard....

i added a service for the node_exporter to expose 9100 and set the Prometheus datasource in grafana to it. I don't think this is what I'm supposed to do, and...

I also created a service for kube-state-metrics, still not the metrics this is looking for. For instance, if I am looking at the "K8s Container" dashboard, the "Total Memory Usage"...

w00t. some progress. As I suspected at the start, in addition to the the kube-state-metrics and node-exporter, you need to manually create the config for a prometheus pod with the...

So far, I have narrowed it down to the prometheus configs for getting cadvisor is not populating, naming is off, or possibly a permissions issue. This is the query from...

yep, permission issue. started prometheus with log-level debug: ```command: ["prometheus","--config.file=/etc/prometheus/prometheus.yml","--log.level=debug"]``` and there are the beautiful 403's. ``` level=debug ts=2019-01-29T05:34:37.90956911Z caller=scrape.go:825 component="scrape manager" scrape_pool=kubernetes-kubelet target=https://kubernetes.default.svc:443/api/v1/nodes/ip-172-22-20-181.us-west-2.compute.internal/proxy/metrics msg="Scrape failed" err="server returned HTTP status...

Oh, this is one of those fun problems that make you question all of your life decisions. This can't be fixed with simple rbac rules, it requires flags set on...

after 3 days of troubleshooting, i unfortunately must concede defeat. if anyone get's this to work on a k8 cluster 1.8+, please do chime in. edit: I CONCEDE NOTHING! Finally...

@illectronic that error comes from this repo https://github.com/grafana/kubernetes-app/blob/ddf616e74c2146e72529316f4fb0348b787e38f4/dist/datasource/datasource.ts#L134 Looks like it's related to the kubernetes api source, but i'm not sure what triggers it exactly.