Better manage CRDs in Helm chart

[JohnStrunk]

Describe the feature you'd like to have. Currently, we provide the VolSync CRDs in the helm/crds directory. CRDs provided here are automatically applied by Helm when it installs a chart, if the CRD does not exist in the cluster. If the CRD exists, the provided files are ignored. Helm's policy re CRDs

Between 0.3 and 0.4, we added a couple of fields to the CRD, but a user that is upgrading from 0.3 will not get the new CRD versions. We need to restructure the chart to get around this limitation.

What is the value to the end user? (why is it a priority?) CRDs will continue to change as the operator evolves, and the installed version must be kept in sync with what the operator expects to see. The alternative to automatic management is requiring the user to manually apply the new/updated CRDs from the appropriate chart version. This is a poor user experience.

How will we know we have a good solution? (acceptance criteria)

• New installs will have the associated CRD installed automatically
• Chart upgrades will apply any CRD changes to bring the CRDs up to date
• Chart uninstall will not remove the CRDs from the cluster

Additional context CRDs can be installed (and templated) just like any other manifest as long as the rest of the Chart doesn't attempt to use the newly installed CRD (and we don't). So here's a potential solution that needs to be validated:

• Move the CRDs to be regular templates
• Wrap the declarations so that they are not removed on uninstall (helm uninstall doesn't remove CRDs)
  • I think this can be done by checking Release.IsUpgrade and Release.IsInstall

[onedr0p]

I think it would be great to follow cert-managers approach to the problem by adding a .Values.installCRDs option to the helm chart and attaching the generated CRDs to the GitHub release. But it would also be nice if you could add them in the repo as well:

deploy/
    crds/
        volsync.backube_replicationdestinations.yaml
        volsync.backube_replicationsources.yaml
        kustomization.yaml # (see below)

# kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - volsync.backube_replicationdestinations.yaml
  - volsync.backube_replicationsources.yaml

This would be very useful to the kubernetes admin to install the CRDs with Kustomize, for example:

kubectl apply -k ./path/to/folder/containing/file/below.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/backube/volsync//deploy/crds?ref=v0.4.0

[JohnStrunk]

My hope is that by installing the Chart, we can get "the right thing" to happen automatically. Today, you can install the CRDs straight from the repo:

$ kubectl apply -k https://github.com/backube/volsync/config/crd?ref=v0.4.0
customresourcedefinition.apiextensions.k8s.io/replicationdestinations.volsync.backube created
customresourcedefinition.apiextensions.k8s.io/replicationsources.volsync.backube created

but that doesn't really help with chart upgrades.

[onedr0p]

As I mentioned in my reply, I would go the route cert-manager did:

https://github.com/cert-manager/cert-manager/blob/f109f34aee45f5c821e89bc4ff65edfe7bdea8be/deploy/charts/cert-manager/values.yaml#L44

And then wrap the crds in a template with that logic. I've seen this pattern on a lot of other charts too like kyverno and rook-ceph.

[JohnStrunk]

/assign