feature: hpa support for vmstorage

[dctrwatson]

This would be really useful to use when properly configured (mainly scaleDown disabled) to no longer require a human to perform the same function as the HPA during regular organic growth of Active Series.

[f41gh7]

I think, it's more useful to have VPA from practical standpoint. Especially, when Kubernetes brings dynamic requests/limits resources feature.

[dctrwatson]

I think, it's more useful to have VPA from practical standpoint. Especially, when Kubernetes brings dynamic requests/limits resources feature.

yes, I agree for when you're not using dedicated nodes for vmstorage. however, you cannot VPA in node-sized steps right now. So you end up wasting a lot of resources or having to continue to share nodes and bin-pack pods that are hopefully not noisy neighbors. Not to mention having to rollout any size changes to all vmstorage replicas.

It's a trade-off to do HPA instead, for us that's made using a single sized vmstorage is much easier to reason about relative scale and operation of various sized vmclusters. Scaling up doesn't require rolling existing vmstorage replicas either.