Exclude NotReady nodes from Node Count

Open morenod opened this issue 4 years ago • 1 comments

When generating the number of pods to create and labeling the nodes, we should exclude NotReady nodes, to avoid generating a number of pods higher to number of schedulable pods

ip-10-0-210-139.us-west-2.compute.internal NotReady,SchedulingDisabled worker 57m v1.23.5+b0357ed

lun 28 mar 2022 12:03:17 UTC Labeling 27 worker nodes with node-density=enabled [...] node/ip-10-0-210-139.us-west-2.compute.internal labeled [...]

lun 28 mar 2022 12:03:47 UTC ############################################### lun 28 mar 2022 12:03:47 UTC Workload: node-density lun 28 mar 2022 12:03:47 UTC Workload template: workloads/node-pod-density/node-pod-density.yml lun 28 mar 2022 12:03:47 UTC Metrics profile: metrics-profiles/metrics.yaml lun 28 mar 2022 12:03:47 UTC Alerts profile: lun 28 mar 2022 12:03:47 UTC QPS: 20 lun 28 mar 2022 12:03:47 UTC Burst: 20 lun 28 mar 2022 12:03:47 UTC Node count: 27 lun 28 mar 2022 12:03:47 UTC Pods per node: 245 lun 28 mar 2022 12:03:47 UTC ###############################################

lun 28 mar 2022 12:03:17 UTC Number of pods to deploy on nodes: 6181

Node count should be 26, not 27

https://github.com/cloud-bulldozer/e2e-benchmarking/blob/86c18427a5004a37e4a3a911fba262f4719ee867/workloads/kube-burner/common.sh#L81

Mar 28 '22 12:03 morenod

I disagree with this as an issue. During our testing we expect all nodes to be ready so we can accurately say we did a node density test on 27 nodes, for example. If nodes are not ready I expect this to fail. Perhaps we could expedite the failure by checking before starting the actual workload but otherwise I don't see an "issue" here IMHO.

Mar 28 '22 15:03 dry923