`vtop-rollout` tooling for safer rollout of new Vitess versions in Kubernetes

[frouioui]

Motivation

While the default behavior of the vitess-operator performs upgrades en masse, there is a desire for more granular control. This issue proposes the introduction of a tool, referred to as vtop-rollout, which will act as a Kubernetes API client similar to kubectl, to address this need.

Definitions

• vtop-rollout is a tool intended to facilitate the planned and orderly release of changes, including Vitess/MySQL component upgrades and configuration modifications (e.g., memory, CPU).
• vtop-rollout will act as a Kubernetes API client.
• vtop-rollout operates using the External updateStrategy of the vitess-operator, enabling it to autonomously manage changes.

More documentation around the updateStrategy is available here: https://github.com/planetscale/vitess-operator/blob/main/docs/api.md#planetscale.com/v2.VitessClusterUpdateStrategy

Example use-case

I have a Vitess cluster running in K8S with the vitess-operator and I want to go from v17.0.2 to v18.0.0 for all my Vitess components. With vanilla vitess-operator all my components will be upgraded at once with no control. If I use vtop-rollout I will be able to control step-by-step the upgrade by selecting which vttablet are upgraded one-by-one, giving me more granular control over my upgrades.

Current process for updateStrategy: External

Setting the updateStrategy in VitessCluster to External leads to no update on the vttablet . Instead the vitess-operator will set an annotation to the resource.

Let’s say we are changing vttablet image tag from v17.0.2 to v18.0.0 here is what we would get in the Annotations field when running kubectl describe VitessKeyspace:

Annotations:  rollout.planetscale.com/scheduled:
              spec:
                images:
                 vttablet: vitess/lite:v18.0.0

How would vtop-rollout work

1. The end-user will change their K8S YAML file and apply it
   1. Change the updateStrategy to External.
   2. Change the Docker Image of the Vitess components from vitess/lite:v17.0.2 to vitess/lite:v18.0.0.
   3. These changes will be applied using kubectl apply -f ....
2. The vitess-operator will not rollout any change to the cluster, instead it will annotate the VitessKeyspace object with the wanted changes.
3. The end-user will then be able to use vtop-rollout tool to begin the rollout.
   1. Using vtop-rollout CLI, the end-user will be able to see all pending changes (that all components should be using vitess/lite:v18.0.0). The pending changes will be grouped by Vitess component and by shards (for vttablets).
   2. Given the list of changes, the end-user will be able to upgrade a single component of Vitess or single Shard, or have an even more granular control by deciding to upgrade only a given tablet in a shard.
   3. When the user decide the upgrade (either one of the three options), vtop-rollout will roll the new Docker Image into the pod using kubectl set image.
   4. The user will then be able to finish the rollout or repeat the few previous steps to upgrade other components/shards.
   5. Finishing the migration will annotate the VitessKeyspace as released using the rollout.planetscale.com/released=true annotation, which will complete any pending actions.

Proposal

1. Development of vtop-rollout Tool: Develop and implement the vtop-rollout tool as a Kubernetes API client for controlled Vitess component rollouts.

2. Documentation Update: Update project documentation to reflect the usage of vtop-rollout in scenarios requiring granular control over Vitess upgrades. This documentation should include clear instructions on how to use vtop-rollout effectively.

3. Clarification of updateStrategy: Clarify the requirement for using the External updateStrategy in conjunction with vtop-rollout. This clarification ensures that the vitess-operator applies the necessary annotations to pods for correct rollout management by vtop-rollout.

The proposed changes aim to enhance the ability to manage Vitess component upgrades in a controlled and orderly fashion, providing users with more flexibility and reliability during the upgrade process.