Heapster (1.3.0) returns message No metrics for pod/unable to get metrics for resource cpu: no metrics returned from heapster

[Mahesh-MSFT]

Is this a request for help?:

---

YES

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes Version: v1.6.6 Heapster Version v1.3.0

What happened: Heapster fails to collect metrics from pods. Heapster pod log message says No metrics for pod

I0809 06:44:08.573960       1 heapster.go:72] /heapster --source=kubernetes.summary_api:""
I0809 06:44:08.574000       1 heapster.go:73] Heapster version v1.3.0
I0809 06:44:08.574346       1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version v1
I0809 06:44:08.574365       1 configs.go:62] Using kubelet port 10255
I0809 06:44:08.575119       1 heapster.go:196] Starting with Metric Sink
I0809 06:44:08.675764       1 heapster.go:106] Starting heapster on port 8082
I0809 06:44:30.404310       1 handlers.go:215] No metrics for pod default/aspnetcoreapp-deployment-1249858457-bx5zl
I0809 06:44:30.404345       1 handlers.go:215] No metrics for pod default/aspnetcoreapp-deployment-1249858457-t6p46
I0809 06:44:30.404349       1 handlers.go:215] No metrics for pod default/aspnetcoreapp-deployment-1249858457-ln4r1

What you expected to happen: Heapster should collect metrics from pods

How to reproduce it (as minimally and precisely as possible): Create a new ACS Linux Cluster with Kubernetes as orchestrator. Create a deployment, service and horizontalpodautoscaler as shown below.

Deployment:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: aspnetcoreapp-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: aspnetcoreapp
    spec:
      containers:
      - name: aspnetcoreapp
        image: maksh/aspnetcorekuberappimg
        ports:
        - containerPort: 5000
        resources:
          requests:
            cpu: 1m

Service

apiVersion: v1
kind: Service
metadata:
  name: aspnetcoreapp-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: aspnetcoreapp

Horizontalpodautoscaler

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: aspnetcoreapp-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: aspnetcoreapp-deployment
  minReplicas: 3
  maxReplicas: 6
  targetCPUUtilizationPercentage: 20

Anything else we need to know: kubectl describe hpa returns following

Name:                                                   aspnetcoreapp-hpa
Namespace:                                              default
Labels:                                                 <none>
Annotations:                                            <none>
CreationTimestamp:                                      Wed, 09 Aug 2017 12:31:13 +0530
Reference:                                              Deployment/aspnetcoreapp-deployment
Metrics:                                                ( current / target )
  resource cpu on pods  (as a percentage of request):   <unknown> / 20%
Min replicas:                                           3
Max replicas:                                           6
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type        Reason
                Message
  ---------     --------        -----   ----                            -------------   --------    ------
                -------
  1m            15s             4       horizontal-pod-autoscaler                       Warning     FailedGetResourceMetric             unable to get metrics for resource cpu: no metrics returned from heapster
  1m            15s             4       horizontal-pod-autoscaler                       Warning     FailedComputeMetricsReplicas        failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster

[lachie83]

Hi @Mahesh-MSFT.

I'm currently unable to repro on a clean 1.6.6 ACS provisioned cluster.

$ kubectl get hpa
NAME                REFERENCE                             TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
aspnetcoreapp-hpa   Deployment/aspnetcoreapp-deployment   0% / 20%   3         6         3          2m

$ kubectl describe hpa
Name:							aspnetcoreapp-hpa
Namespace:						default
Labels:							<none>
Annotations:						<none>
CreationTimestamp:					Thu, 10 Aug 2017 15:27:16 -0700
Reference:						Deployment/aspnetcoreapp-deployment
Metrics:						( current / target )
  resource cpu on pods  (as a percentage of request):	0% (0) / 20%
Min replicas:						3
Max replicas:						6
Events:							<none>

[lachie83]

I have seen it take some time initially to pull the stats.

[Mahesh-MSFT]

Have given it more that 30 mins but still no cpu stats.

kubectl describe hpa
Name:                                                   aspnetcoreapp-hpa
Namespace:                                              default
Labels:                                                 <none>
Annotations:                                            <none>
CreationTimestamp:                                      Fri, 11 Aug 2017 16:46:18 +0530
Reference:                                              Deployment/aspnetcoreapp-deployment
Metrics:                                                ( current / target )
  resource cpu on pods  (as a percentage of request):   <unknown> / 20%
Min replicas:                                           3
Max replicas:                                           6
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type        Reason
                Message
  ---------     --------        -----   ----                            -------------   --------    ------
                -------
  42m           10s             86      horizontal-pod-autoscaler                       Warning     FailedGetResourceMetric             unable to get metrics for resource cpu: no metrics returned from heapster
  42m           10s             86      horizontal-pod-autoscaler                       Warning     FailedComputeMetricsReplicas        failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster

kubectl get hpa
NAME                REFERENCE                             TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
aspnetcoreapp-hpa   Deployment/aspnetcoreapp-deployment   <unknown> / 20%   3         6         3          44m

I am using az acs create --orchestrator-type kubernetes --resource-group myResourceGroup --name myK8sCluster --ssh-key-value="blahblah" and then connecting from kubectl on local machine after kubectl proxy

Are you following same steps? Does Reource Group location makes any difference? My rg is in southeastasia.

[lachie83]

I don't believe so. Let me try an repro in that region

[lachie83]

I've just tested in the southeastasia region and I'm seeing the heapster deployment being updated in a loop. I believe it's the same root cause as https://github.com/Azure/acs-engine/pull/1133

Temporary work around is to add that line to /etc/kubernetes/addons/kube-heapster-deployment.yaml on all masters.

[Mahesh-MSFT]

Yes. You suggested workaround has solved the problem. My steps as below -

1. Note MASTERFQDN and LINUXADMINUSERNAME from "deployment" pane of resource group.
2. Log onto MASTERFQDN using Putty.
3. Run pwd to determine current working directory.
4. Change directory to /etc/kubernetes/addons
5. Run sudo vim kube-heapster-deployment.yaml
6. Add line addonmanager.kubernetes.io/mode: EnsureExists as specified here
7. Save and exit vim
8. Run kubectl get hpa
9. Verify Targets should display current and target CPU utilization

[lachie83]

@Mahesh-MSFT We are working on getting the fix rolled out. Let's keep this open until it's been fixed in ACS

[mfaizanse]

Is there any update for heapster with windows containers?

[hmarcelodn]

I have heapster running as explained by @Mahesh-MSFT but I am still experiencing de unknown message for current resources. Any other way to try that ?

[samlyhuang]

I have the same problem kubernetes version:1.6.2 heapster version:1.3.0

kubectl get deployment                    
NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
ceph-pod5-with-secret   2            2                   2                  2           10h

kubectl get pods                          
NAME                                     READY     STATUS    RESTARTS   AGE
ceph-pod5-with-secret-3074552501-6ffqd   1/1       Running   0          10h
ceph-pod5-with-secret-3074552501-sqq6k   1/1       Running   0          10h

 kubectl describe hpa ceph-pod5-with-secret
Name:                                                   ceph-pod5-with-secret
Namespace:                                              default
Labels:                                                 <none>
Annotations:                                            <none>
CreationTimestamp:                                      Sat, 04 Nov 2017 21:22:52 +0800
Reference:                                              Deployment/ceph-pod5-with-secret
Metrics:                                                ( current / target )
  resource cpu on pods  (as a percentage of request):   <unknown> / 5%
Min replicas:                                           2
Max replicas:                                           5
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                            -------------   --------        ------                          -------
  10h           16s             98      horizontal-pod-autoscaler                       Warning         FailedGetResourceMetric         unable to get metrics for resource cpu: no metrics returned from heapster
  10h           16s             98      horizontal-pod-autoscaler                       Warning         FailedComputeMetricsReplicas    failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster

[hmarcelodn]

@samlyhuang it should work if you enable heaspter in the cluster and apply resources/limits to the manifest for the container. did you try that ?

[samlyhuang]

@hmarcelodn I have enabled heaspter in the cluster.When I open kubernetes dashboard,I can find CPU usage and Memory usage,so I think heaspter work well.

my yaml file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ceph-pod6-with-secret
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: ceph-pod6-with-secret
    spec:
      containers:
      - name: ceph-ubuntu4
        image: ubuntu:14.04
        resources:
          limits:
            cpu: 150m
            memory: 50Mi
          requests:
            cpu: 150m
            memory: 50Mi
        command: ["tail", "-f", "/var/log/bootstrap.log"]
        volumeMounts:
        - name: ceph-fs2
          mountPath: /mnt
          readOnly: false
      volumes:
      - name: ceph-fs2
        cephfs:
          monitors:
          - 192.168.31.150:6789
          user: admin
          secretFile: "/etc/ceph/admin.secret"
          readOnly: false
        #ports:
        #- containerPort: 80

do you find any error configs?

[diwakar-s-maurya]

@samlyhuang Did you find solution to your problem? I am also facing the same problem.

[LakshyaGarg]

I am facing the same issue, any solution for this ?

[abhikbanerjee]

I am having the same issue, any solution for the same, I have enabled heapster, but still get the same message

[Mahesh-MSFT]

@samlyhuang @LakshyaGarg @abhikbanerjee temporary workaround as described here has resolved my issue.

[LakshyaGarg]

@abhikbanerjee I solved this by checking out the latest heapster from https://github.com/kubernetes/heapster and running it solved the issue.