oci-cloud-controller-manager icon indicating copy to clipboard operation
oci-cloud-controller-manager copied to clipboard
verified publisher icon oracle

Can not add NLB backend automatically after create load balancer service.

Open alex-wong123 opened this issue 3 years ago • 4 comments

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

Versions

CCM Version: V1.23.0 Environment:

  • Kubernetes version (use kubectl version): v1.23.4
  • OS (e.g. from /etc/os-release): Oracle Linux Server 7.9
  • Kernel (e.g. uname -a): 5.4.17-2136.309.5.el7uek.x86_64
  • Others:

What happened?

Created a load balancer service successfully, and then a NLB was created automatically. But access the service failed, then check the NLB, there was no member in backend.

What you expected to happen?

The load balancer service can be accessed successfully.

How to reproduce it (as minimally and precisely as possible)?

CCM was installed as doc

Anything else we need to know?

The logs of oci-cloud-controller-manager pod using 'kubectl logs oci-cloud-controller-manager-7lblr' as bellow 2022-11-16T14:27:10.622Z INFO oci/load_balancer.go:849 Updating load balancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.622Z INFO oci/load_balancer.go:409 Ensuring load balancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "nodes": 3} 2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node3" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node1" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node2" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.664Z INFO oci/load_balancer_security_lists.go:96 Security list management mode: "None". Not managing security lists. {"component": "cloud-controller-manager"} 2022-11-16T14:27:10.664Z INFO oci/load_balancer.go:543 Successfully updated loadbalancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} I1116 14:27:10.664509 1 controller.go:748] Successfully updated 1 out of 1 load balancers to direct traffic to the updated set of nodes I1116 14:27:10.664593 1 event.go:294] "Event occurred" object="default/nginx-nlb-svc" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated load balancer with new hosts" E1116 14:29:34.871159 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:29:35.056435 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:29:35.256508 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:34:36.270177 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:34:36.441161 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:34:36.705268 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:39:37.425184 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:39:37.551867 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:39:37.642478 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:44:38.310129 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:44:38.552157 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:44:38.776831 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID

alex-wong123 avatar Nov 16 '22 15:11 alex-wong123

It looks like you are not setting the provider ID during Kubelet initialization. Please see the kubelet docs for adding provider ID via command line arg or kubelet config file https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ image

AkarshES avatar Nov 16 '22 16:11 AkarshES

The flags are set for kube-controller-manager and kubelet ordering the docs. root 4586 0.8 0.7 826256 118492 ? Ssl 14:30 4:16 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cloud-provider=external --cluster-cidr=10.246.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.98.0.0/16 --use-service-account-credentials=true root 20483 2.2 0.7 1783240 121780 ? Ssl 14:27 12:10 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --provider-id=ocid1.instance.oc1.us-sanjose-1.anzwuljry5l3z3ycjdgcnhctmu3aljxjhki2oglgjmqd75ymzpzxmnudeg3a --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.4.1

alex-wong123 avatar Nov 16 '22 23:11 alex-wong123

+1

keinou avatar Feb 20 '23 00:02 keinou

node_controller.go takes ProviderID from a node spec (https://github.com/kubernetes/cloud-provider/blob/master/controllers/node/node_controller.go#L663C30-L663C30). After adding ProviderID to spec for each node the "Error getting instance metadata for node addresses" errors were gone and the backends were added

4kord avatar Dec 07 '23 03:12 4kord