Package version maintenance best practice

[leonard520]

Hi team, I know there are some concepts to maintain the package versions, PackageRepository and PackageInstall. PackageRepository can include several Packages and each Package can have several versions. Assume we have already install PackageRepository my-repo:1.0.0 and the PackageInstall my-package:0.0.1 in the cluster. when there are my-package:0.0.2 released, we will have the below options.

1. Create a new my-repo:1.0.1 with my-package:0.0.2
2. Add my-package:0.0.2 to my-repo:1.0.0
3. Create a new my-repo:1.0.1 with my-package:0.0.1 and my-package:0.0.1

If it is the first case, to upgrade, we need to add my-repo:1.0.1 to k8s cluster, and then update PackageInstall from 0.0.1 to 0.0.2 If it is the second case, we just need to push the new package repo 1.0.0 to the container registry and then update PackageInstall from 0.0.1 to 0.0.2. we don’t need to install the new PackageRepository If it is the third case, I am afraid we have to remove my-repo:1.0.1 from k8s cluster first because there is already Package named my-package:0.0.1 in the cluster. The new added my-repo:1.0.1 won’t be able to reconcile if my-repo:1.0.0 still exists since both of them contains my-package:0.0.1. I am worried the removal of old my-repo:1.0.0 and install of the new version will bring some trouble for the existing PackageInstall as there will be a while no PackageRepository at all in the cluster.

[joe-kimmel-vmw]

Thanks @leonard520 - as of kapp-controller 0.38.x this sentence is false (as long as the two packages with identical name+version are completely identical):

The new added my-repo:1.0.1 won’t be able to reconcile if my-repo:1.0.0 still exists since both of them contains my-package:0.0.1.

Because of that the short term recommendation is to leverage this ability and to prefer a path where you make one package repository per version. Does that help? We're definitely interested to hear more about your use-case and understand what other features/behaviors you might need.

[joe-kimmel-vmw]

see the docs here: https://carvel.dev/kapp-controller/docs/v0.38.0/packaging/#overlapping-package-repositories

[cppforlife]

to add a bit more... all three methods are supported and can be useful depending on styles of usage. we typically recommend either:

• (a) to keep on adding packages to existing repo (most common recommendation)
• or, (b) having 1 pkg-repo == 1 "software" version (typically software is composed of multiple packages).

in case of (a) the only difficulty if you have to transfer pkg repo to a private registry -- it might be quite large if you have a lot of packages in there and currently the only solution is to periodically prune older packages out.

in case of (b) one can either upgrade existing pkg repo CR to point to new pkg-repo or add a new CR along side (this is what joe is referring to with kc v0.38.x changes).

I am worried the removal of old my-repo:1.0.0 and install of the new version will bring some trouble for the existing PackageInstall as there will be a while no PackageRepository at all in the cluster.

kapp-controller is very conservative around this use case. if PackageInstall fails to find referenced Package, it will just remain in ReconcileFailed step until it finds that Package or until it's modified to point to new Package. kc also disallows downgrades by default so your PackageInstall wont just go back to earlier versions if you are using loose versionSelection constraint (e.g. always use latest version).

[leonard520]

@cppforlife Thanks for the suggestions. I have tried to use Kapp controller v0.38 and it can work with overlap package versions of different package repos. It is really cool. However, I meet an issue and want to double confirm with you. Say I have my-repo:1.0.0 which contains my-package:0.0.1 and your-package:0.0.1 I apply them in my cluster and now I am using my-package:0.0.1 and your-package:0.0.1 running. One day, your-package has upgrade to your-package:0.0.2. I decide to use make a new package repo my-repo:2.0.0 to include my-package:0.0.1 and your-package:0.0.2. I add my-repo:2.0.0 to my cluster. Now I have package my-package:0.0.1 and your-package:0.0.1 and your-package:0.0.2 I upgrade the existing your-package to 0.0.2 to let user enjoy the latest feature. Then I want to remove my-repo:1.0.0 to keep the cluster more clean -- my-repo:1.0.0 is not needed any more. In my testing, I find the removal of my-repo:1.0.0 will delete the associate package, too. It will remove my-package:0.0.1 as well even if my-repo:2.0.0 are also using it. It won't impact existing my-package indeed but the new creation of packageinstall my-package:0.0.1 will stuck reconcile failed until my-repo:2.0.0 reconcile first. The default sync time should be 10minutes. It is a bit longer. Even if it is configurable, I am wondering if the removal of package repo can be refined to delete those not be referred.

[cppforlife]

@leonard520 you are right that this is indeed the current behaviour. when we were implementing it we didnt see an easy way to make this work quicker. definitely need to consider how we can improve this.

[kvmw]

to add a bit more... all three methods are supported and can be useful depending on styles of usage. we typically recommend either:

• (a) to keep on adding packages to existing repo (most common recommendation)
• or, (b) having 1 pkg-repo == 1 "software" version (typically software is composed of multiple packages).

in case of (a) the only difficulty if you have to transfer pkg repo to a private registry -- it might be quite large if you have a lot of packages in there and currently the only solution is to periodically prune older packages out.

in case of (b) one can either upgrade existing pkg repo CR to point to new pkg-repo or add a new CR along side (this is what joe is referring to with kc v0.38.x changes).

I am worried the removal of old my-repo:1.0.0 and install of the new version will bring some trouble for the existing PackageInstall as there will be a while no PackageRepository at all in the cluster.

kapp-controller is very conservative around this use case. if PackageInstall fails to find referenced Package, it will just remain in ReconcileFailed step until it finds that Package or until it's modified to point to new Package. kc also disallows downgrades by default so your PackageInstall wont just go back to earlier versions if you are using loose versionSelection constraint (e.g. always use latest version).

to clarify one point about option a, We need a new version of the package-repo, every time that we are adding/updating the packages. we cannot edit the packages in the existing package repo without creating a new image of it. am i right?

[cppforlife]

We need a new version of the package-repo, every time that we are adding/updating the packages. we cannot edit the packages in the existing package repo without creating a new image of it. am i right?

right (registry resources are immutable), but you could be moving a tag to reference latest copy.

[leonard520]

Thanks @kvmw for the clarification. It is clear now. @cppforlife I have one further question regarding to PackageRepo. Since registry resources are immutable, say my-repo:1.0.0 always have my-package:0.0.1 and my-repo:1.0.0 will have my-package:0.0.1 and my-package:0.0.2, what's the purpose for reconcile? Does it make sense to give the syncPeriod a large number? Also, the question is similar to PackageInstall if the package behind is immutable for a certain version. Do I miss some cases?

[joe-kimmel-vmw]

@leonard520

Since [...] resources are immutable [...] what's the purpose for reconcile?

In the case everything you're working with is completely fixed, you can think of it as assurance that the cluster is in the desired state. Any manual or automated changes not reflected in the declarative resources will be wiped clean at a regular interval, so it means that the system can be expected to be in the declared state. In my head it's a little similar to how some providers will do a forced restart of a VM every so often which forces the developers to ensure that everything boots cleanly and doesn't rely on some manually set one-time config.

In general kapp-controller doesn't always require things to be completely fixed and the mechanisms are more general. For instance you can set a PackageInstall to prefer a range of packages and effectively install security patches automatically as they become available through new (or updated) package Repositories.

[neil-hickey]

I'm not 100% sure if this issue is ok to be closed or not, if I don't hear anything within a week or two I will close it out. Thanks for the discussions!

[leonard520]

Thanks for the great discussion! I am closing the issue.