Add podAntiAffinity when vmagent in shard mode

[okzheng]

We have three k8s pod for deploying vm components. Then we config a vmagent cr with 2 shards, every shard with 2 replicas. When I add podAntiAffinity config for vmagent in shard mode, I could only add some labels to all sharding, and I could not add label shardNum. The result is all vmagent pods from a vmagent cr will be anti affinity. It is unnecessary avoid placing vmagents from different shard on a node that match same conditions. We just need to avoid scheduling vmagents pod from a same shard to a node. So if the vm-operator could add label shardNum to podAntiAffinity automatically or provide a flag to control the action, It would fit in more places. May be have more reasonable resolution?

[f41gh7]

Hello, operator adds shard-num label to the podSpec:

spec:
  template:
      metadata:
         labels:
            shard-num: "1"

but it isn't enough to create proper antiAffinity rule. Need to think, how to implement it properly. Maybe some boolean flag, that will replace value for shard-num label with correct number at affinity section.

[f41gh7]

Since v0.31.0 VMAgent supports placeholder for %SHARD_NUM%. It could be useful for affinity and pdb generation.

Need to add docs for it.

[hagen1778]

The doc is updated here https://docs.victoriametrics.com/operator/high-availability.html?highlight=affinity#sharding