Kapp-Controller fails to install package when using versionSelection constraints AND a packageRepository

[davidblum]

What steps did you take:

Steps

Create and push Bundle Create and push PackageRepository Attempt to install package via package repository Error

For this example we are using KIND + a local docker registry (with no auth)

Create and push bundle

We use Istio's basic httpbin example for our test service.

mkdir -p dist/service dist/.imgpkg

curl https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml -o dist/service/http-bin.yml

kbld -f dist/service --imgpkg-lock-output dist/.imgpkg/images.yml

setup latest and 0.1.0

imgpkg push -b kind-registry.local:5001/http-bin:latest -f dist

imgpkg push -b kind-registry.local:5001/http-bin:0.1.0 -f dist

Create and push packageRepository

❯ tree package-repository
package-repository
├── .imgpkg
└── packages
    └── http-bin
        ├── 0.1.0.yml
        └── metadata.yml

Package:

❯ cat package-repository/packages/http-bin/0.1.0.yml
 # See https://carvel.dev/kapp-controller/docs/v0.36.1/packaging/#package

  apiVersion: data.packaging.carvel.dev/v1alpha1
  kind: Package
  metadata:
    name: example.com.http-bin.0.1.0
    namespace: kapp-controller-packaging-global
  spec:
    refName: example.com.http-bin
    version: 0.1.0
    releaseNotes: "install install istio http-bin"
    template:
      spec:
        fetch:
        - imgpkgBundle:
            image: kind-registry.local:5000/http-bin
            tagSelection:
              semver:
                constraints: 0.1.0
        template:
          - kbld:
              paths:
                - /service
        deploy:
          - kapp: {}

Package Meta:

❯ cat package-repository/packages/http-bin/metadata.yml
  # See https://carvel.dev/kapp-controller/docs/v0.36.1/packaging/#package-metadata

  apiVersion: data.packaging.carvel.dev/v1alpha1
  kind: PackageMetadata
  metadata:
    name: example.com.http-bin
    namespace: kapp-controller-packaging-global
  spec:
    displayName: "you fill this out"
    longDescription: "Negative, I am a meat popsicle"
    shortDescription: "Lilu Dallas, multi-pass"
    maintainers:
      - name: test-team
    categories:
      - "test"
      - "http-bin"
    supportDescription: "Don't @ me"

Build package repository bundle

kbld -f ./package-repository/packages --imgpkg-lock-output "./package-repository/.imgpkg/images.yml"

Push package repository to registry

imgpkg push -b kind-registry.local:5001/package-repository:latest -f package-repository

imgpkg push -b kind-registry.local:5001/package-repository:0.1.0 -f package-repository

Install package repository

❯ cat package_repository_install.yml
  apiVersion: packaging.carvel.dev/v1alpha1
  kind: PackageRepository
  metadata:
~   name: package-repository.example.com
    # Adds it to global namespace (as defined by kapp-controller)
    # which makes packages available in all namespaces
    namespace: kapp-controller-packaging-global
  spec:
    fetch:
      imgpkgBundle:
        image: kind-registry.local:5000/package-repository:0.1.0

kubectl apply -f package_repository_install.yml

Install package

❯ cat package_install.yml
  apiVersion: packaging.carvel.dev/v1alpha1
  kind: PackageInstall
  metadata:
    name: http-bin
    namespace: default
  spec:
    serviceAccountName: kapp-controller-admin
    packageRef:
      refName: example.com.http-bin
      versionSelection:
        constraints: 0.1.0

kubectl apply -f package_install.yml

Error

Error on packageInstall:

❯ kubectl get pkgi http-bin -o jsonpath="{.status.usefulErrorMessage}"
vendir: Error: Syncing directory '0':
  Syncing directory '.' with imgpkgBundle contents:
    Imgpkg: exit status 1 (stderr: imgpkg: Error: Checking if image is bundle: could not parse reference: kind-registry.local:5000/http-bin@sha256:82cbaedef6df9033cf4d78a7dd12080f8c661a8e662bae2117f2e25551d90e4a:0.1.0

Full output

❯ k get pkgi http-bin -o yaml
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"packaging.carvel.dev/v1alpha1","kind":"PackageInstall","metadata":{"annotations":{},"name":"http-bin","namespace":"default"},"spec":{"packageRef":{"refName":"example.com.http-bin","versionSelection":{"constraints":"0.1.0"}},"serviceAccountName":"kapp-controller-admin"}}
  creationTimestamp: "2022-06-06T19:45:50Z"
  finalizers:
  - finalizers.packageinstall.packaging.carvel.dev/delete
  generation: 1
  name: http-bin
  namespace: default
  resourceVersion: "3896"
  uid: cc267fae-3082-4853-bd79-2bbbfc346eaf
spec:
  packageRef:
    refName: example.com.http-bin
    versionSelection:
      constraints: 0.1.0
  serviceAccountName: kapp-controller-admin
status:
  conditions:
  - message: Error (see .status.usefulErrorMessage for details)
    status: "True"
    type: ReconcileFailed
  friendlyDescription: 'Reconcile failed: Error (see .status.usefulErrorMessage for
    details)'
  lastAttemptedVersion: 0.1.0
  observedGeneration: 1
  usefulErrorMessage: |
    vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Imgpkg: exit status 1 (stderr: imgpkg: Error: Checking if image is bundle: could not parse reference: kind-registry.local:5000/http-bin@sha256:82cbaedef6df9033cf4d78a7dd12080f8c661a8e662bae2117f2e25551d90e4a:0.1.0
    )
  version: 0.1.0

What happened: The package failed to install due to an non-parseable image location string

What did you expect: The package to install succesfully

Anything else you would like to add:

You can work around this issue by omitting:

      versionSelection:
        constraints: 0.1.0

And place the OCI image tag on the refName object

      refName: example.com.http-bin:0.1.0

Environment:

• kapp Controller version (execute kubectl get deployment -n kapp-controller kapp-controller -o yaml and the annotation is kbld.k14s.io/images):
• Kubernetes version (use kubectl version)

kapp-controller image: image": "ghcr.io/vmware-tanzu/carvel-kapp-controller@sha256:c48ee804fb628c18bbdb7010c93c6e9f9995f559843c7de4ec7922d6e64ef055" kube version:

❯ k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-03-06T21:32:53Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[cppforlife]

hey @davidblum, the reason for this behaviour is because imgpkg bundle fetching (done by vendir) does not support (intentionally) using digest and tagSelection: https://github.com/vmware-tanzu/carvel-vendir/blob/develop/pkg/vendir/fetch/imgpkgbundle/sync.go#L98-L100 (though we do not have a nice error message for this).

but where does the digest come from? it ends up to be added via kbld command that runs as part of PackageRepository CR (this step kbld -f ./package-repository/packages --imgpkg-lock-output "./package-repository/.imgpkg/images.yml" also locks down which digest should be used).

so ultimately we end up with following... (below snippet) ...which raises a question, what should happen in this case: digest and tagSelection specified?

  spec:
    template:
      spec:
        fetch:
        - imgpkgBundle:
            image: kind-registry.local:5000/http-bin@sha256:abc123...
            tagSelection:
              semver:
                constraints: 0.1.0

im guessing because you were setting constraints: 0.1.0 you want this Package to point to exactly specific artifact. if so, i would just use image: kind-registry.local:5000/http-bin:0.1.0 (with no tagSelection, since you do not want to select tags more dynamically). kbld will replace 0.1.0 with digest and everything will be ok.

[davidblum]

@cppforlife Thanks for the reply!

This is exactly what we ended up doing.

From what we read in the documentation our approach was supported, so we were very confused when it resulted in the aforementioned error.

I think either a clearer error message, or perhaps cleaning up that aspect of the documentation could help avoid this error in the future.

[neil-hickey]

closing this out, I think we were able to get a solution to your specific problem on this one @davidblum !