kubeblocks icon indicating copy to clipboard operation
kubeblocks copied to clipboard

[BUG] Deadlock, when disabling an addon if there is any cluster refers to it

Open shanshanying opened this issue 1 year ago • 2 comments

version

kbcli version
Kubernetes: v1.28.3-aliyun.1
KubeBlocks: 0.9.1-beta.13
kbcli: 0.9.0

How to reproduce:

  1. enable an addon on
  2. create a cluster
  3. disable the addon
  4. delete the cluster

We noticed that the cluster cannot be deleted because the the referenced ClusterDefinition is not up to date Check the referred Clusterdefinition and we found the

  • metadata.generation is 3
  • status.observeredGenerations is 2 and the CD won't be deleted cause there is a cluster refering to it.

Comparing the CD before and after disable Addon, we found the generation is increased from 2 to 3. img_v3_02dp_17ee29a6-8df5-48b8-9761-79d03041028g

A quick workaround is

  • patch the CD's status, e.g. k patch cd milvus-2.3.2 -p'{"status":{"observedGeneration":3}}' --type=merge --subresource status

But we should shoot the root cause first to break the deadlock.

shanshanying avatar Aug 15 '24 13:08 shanshanying

If you upgrade the addon

kbcli addon upgrade milvus --version=0.9.0

The operation appears successful, and the addon shows the new version.

However, upon inspection of the ClusterDefinition custom resource (CR), it remains in a deleted state and still displays the old 0.8 version.

weicao avatar Aug 15 '24 14:08 weicao

In the version 1.0.0-alpha.5 and onwards, if the current addon has dependent clusters, any attempts to delete or disable the addon will be intercepted with a warning until you remove the corresponding dependent clusters. if you try execute : kubectl delete addon XXX or kubectl patch addon XXX --type='merge' -p '{"spec":{"install":{"enabled":false}}} or other command to delete or disable the addon , you man see: 0CGrFJooDn

skyrise-l avatar Sep 04 '24 11:09 skyrise-l