Support scaling in multiple TiKV/TiFlash instances simultaneously

[DanielZhangQD]

Feature Request

Is your feature request related to a problem? Please describe:

Currently, during scaling in, TiDB Operator scales in the TiKV and TiFlash one by one, actually, multiple instances can be scaled simultaneously, however, scaling in too many instances at the same time may cause a performance drop because there are too many region replicas are being rescheduled.

Describe the feature you'd like:

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[DanielZhangQD]

We may add a new field spec.tikv.maxScaleInReplicas to control this behavior, default to 1, which is the same as the current behavior.

[lizhemingi]

I would like to help with this one if you don't mind.

[DanielZhangQD]

@duduainankai Sure, thanks a lot!

[lizhemingi]

@DanielZhangQD Hi, here is what I going to do:

• scale in multi tikv store in one ScaleIn call.

• In order to do that find as many as possible but no more than spec.tikv.maxScaleInReplicas in one another scaleOne like call, maybe scaleMulti.

• Try every possible ordinal found and aggregate all error to return.

Please point out if something is wrong.