Fix AKS Scaling (Cluster Autoscaler)

Open mithunshanbhag opened this issue 2 years ago • 0 comments

We're investigating the following options for AKS scaling:

1. ACI VIRTUAL NODES

Currently BLOCKED.

The code changes are in mithun/hpa2 branch. (see PR microsoft/ContosoTraders#78)
The deployment is in the contoso-traders test subscription.

Redeployed AKS cluster via bicep template from mithun/hpa2 branch, which has the Azure CNI network policy (instead of the default kubenet policy).
Had to manually modify AKS's vnet to create a new subnet aci-subnet with address space 10.255.0.0/16.
Tethered it to existing AKS cluster using az aks enable-addons (full instructions here).
Applied the Deployment.yaml manifest from mithun/hpa2 branch, which has the nodeSelector, tolerations changes to configure pods to only run in virtual nodes.

The pods (configured to run in ACI virtual nodes) are stuck in waiting state.

The logs only show that an active endpoint is not being detected for the services / ingress

Could have something to do with the fact that we switched over to Azure CNI network policy instead of the default kubenet policy.
Could have something to do with the nodeSelector, tolerations changes made in the Deployment.yaml file to configure pod to only run in virtual nodes.

Currently INVESTIGATING

The changes are in my fork in [mithun/enable-autoscal](mithunshanbhag:mithun/cluster-autoscaler) branch (See PR microsoft/ContosoTraders#81)
Deployed in Jithin's MSDN subscription.

Load test has a high failure rate. This issue is being tracked separately in microsoft/Contoso-Traders-Cloud-Testing#3
The pods are also not scaling out (this could be related to above issue).

Currently none, still investigating.

Ingress controller was stuck in PENDING state for a few minutes after provisioning. Then automatically went to OK state.

Mar 07 '23 15:03 mithunshanbhag