clickhouse-operator icon indicating copy to clipboard operation
clickhouse-operator copied to clipboard

After upgrading to 0.17, an exception error occurs when modifying clickhouseinstallation

Open czhfe opened this issue 4 years ago • 12 comments

After upgrading to 0.17 an exception error occurs when modifying clickhouseinstallation, which prevents modification and deletion of clickhouseinstallation. the error is as follows:

The ClickHouseInstallation "clickhouse" is invalid: status.errors: Invalid value: "null": status.errors in body must be of type array: "null"

czhfe avatar Dec 22 '21 05:12 czhfe

How exactly do you upgrade to 0.17? Did you upgrade crds or just update image in deployment?

Could you share the results for following command?

kubectl get deploy --all-namespaces | grep clickhouse-operator

Slach avatar Dec 22 '21 08:12 Slach

How exactly do you upgrade to 0.17? Did you upgrade or just update image in deployment?crds

Could you share the results for following command?

kubectl get deploy --all-namespaces | grep clickhouse-operator

Upgrade operator first, then upgrade crd.

image

czhfe avatar Dec 22 '21 08:12 czhfe

Could you upgrade CRDs first your deployment not look like after standard installation way it should be clickhouse-operator instead of clickhouse-operator-altinity-clickhouse-operator? Did you use some Helm chart or something else?

Could you remove this deployment (it safe)

kubecetl delete deploy -n kube-system clickhouse-operator-altinity-clickhouse-operator

after it and just apply

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

and check results

kubectl get deploy clickhouse-operator -n kube-system 

Slach avatar Dec 22 '21 09:12 Slach

Could you upgrade CRDs first your deployment not look like after it should be clickhouse-operator instead of ? Did you use some Helm chart or something else?standard installation way``clickhouse-operator-altinity-clickhouse-operator

Could you remove this deployment (it safe)

kubecetl delete deploy -n kube-system clickhouse-operator-altinity-clickhouse-operator

after it and just apply

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

and check results

kubectl get deploy clickhouse-operator -n kube-system 

I wrote the helm chart deployment myself

czhfe avatar Dec 22 '21 09:12 czhfe

@czhfe ok. try update crds in your helm chart and re-install clickhouse-operator

Slach avatar Dec 22 '21 09:12 Slach

@czhfe ok. try update crds in your helm chart and re-install clickhouse-operator

Now I do not understand why the operator and crd updated here will error, there is no solution

czhfe avatar Dec 22 '21 10:12 czhfe

It solution cause new CRDs define status as subresources.

Compare your current CRDs with https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

Slach avatar Dec 22 '21 10:12 Slach

It solution cause new CRDs define status as subresources.

Compare your current CRDs with https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

I upgraded from 0.13.5 to 0.17

czhfe avatar Dec 22 '21 10:12 czhfe

Upgrade from clickhouse-operator 0.13.5 to 0.17.0. perform the following steps.

  1. Upgrade crd first
  2. Upgrade operator and other resources

After the upgrade, modify ClickHouseInstallation CR encountered the following error

image

czhfe avatar Dec 29 '21 06:12 czhfe

Do you have status in your edited resource? If yes, please remove this sections and try to save

Could you share the following command results?

kubectl get chi -n ch1 clickhouse -o yaml

Slach avatar Dec 29 '21 08:12 Slach

Do you have in your edited resource? If yes, please remove this sections and try to savestatus

Could you share the following command results?

kubectl get chi -n ch1 clickhouse -o yaml

Yes, I have tested this before by deleting the status section and trying to save it later, but the result is still the same.

czhfe avatar Dec 30 '21 05:12 czhfe

Problem recurrence:

The /spec/templating/policy in the clickhouseinstallations.clickhouse.altinity.com CRD prior to version 0.16.0 did not have its default value explicitly defined, resulting in that default value being "". The clickhouseinstallations.clickhouse.altinity.com CRD was upgraded in version 0.16.0 of the clickhouse operator (this can be seen in the release notes in RELEASES). In version 0.16.0 and later /spec/templating/policy was changed to be configurable as auto or manual, with the default value being manual. clickhouseinstallations.clickhouse.altinity.com before 0.16.0 CR before 0.16.0 did not explicitly define that the /spec/templating/policy field would be added automatically, so clickhouse operators and CRD versions after 0.16.0 cannot automatically update the /spec/templating/policy field, which would block clickhouse operator to update the CR for clickhouseinstallations.clickhouse.altinity.com, which causes an error

czhfe avatar Dec 30 '21 05:12 czhfe

Should be fixed in earlier released, but additional rules were added at https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.22.1

alex-zaitsev avatar Nov 30 '23 14:11 alex-zaitsev