Modify autoscaler 'scale-down-utilization-threshold'

[montyleew]

My customers would like to modify the autoscaler 'scale-down-utilization-threshold' to more aggressively scale down nodes when the load decreases. The default is currently 50% and we see that the load decreases, but stays above 50% utilization, where it could have scaled down to remove a node.

The 'scale-down-utilization-threshold', similar to AKS, is not an option in ARO, but we request it to be added to allow us to tune how the cluster scales down.

The request threshold on a particular node should also not be the only criteria for a scale-down. I propose adding a logic to the Autoscaler that will calculate the number of needed nodes and then mark superfluous nodes for scale down even if their requested cores are above the utilization threshold, allowing the pods to be scheduled on the remaining nodes and thus the nodes are used more effectively, and efficiently.

This would all depend on the user being able to set the scale down threshold to suit their application.

[jboutaud]

Thanks for the request. We are Looking at adding this as a feature into OpenShift so that it will make it to ARO.