skaffold icon indicating copy to clipboard operation
skaffold copied to clipboard

panic: interface conversion: runtime.Object is nil, not *v1.Pod

Open mjhuber opened this issue 1 year ago • 0 comments

Sporadically receive the error panic: interface conversion: runtime.Object is nil, not *v1.Pod when running skaffold build.

Expected behavior

  1. Run skaffold build
  2. Container is built and pushed.

Actual behavior

  1. Run skaffold build.
  2. Approximately 5% of the time, skaffold fails, e.g.,
Building [us-docker.pkg.dev/recharge-infra-services-8b42/gcr.io/internal-admin-tools]...
panic: interface conversion: runtime.Object is nil, not *v1.Pod
goroutine 200 [running]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.WaitForPodInitialized.func1(0xc0015cee98?)
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/kubernetes/wait.go:111 +0xf2
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.watchUntilTimeout({0x3e28b30?, 0xc000b78600?}, 0x0?, {0x3e0c7c8, 0xc00137c280}, 0xc0015cefd0)
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/kubernetes/wait.go:49 +0x170
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.WaitForPodInitialized({0x3e28b30, 0xc000b78600}, {0x3e432b0, 0xc0000dd020}, {0xc001162cf0, 0xc})
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/kubernetes/wait.go:110 +0x236
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build/cluster.(*Builder).setupKanikoBuildContext(0xc0006185b0, {0x3e28b30, 0xc000b78600}, {0x5589a70, 0x1}, {0xc0009b4af0, 0x4a}, 0xc0008143c0, {0x3e432b0, 0xc0000dd020}, ...)
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/cluster/kaniko.go:169 +0x88
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build/cluster.(*Builder).buildWithKaniko(0xc0006185b0, {0x3e28b30, 0xc000b78600}, {0x3df6c80, 0xc0005ee780}, {0x5589a70, 0x1}, {0xc0009b4af0, 0x4a}, 0xc0008143c0, ...)
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/cluster/kaniko.go:104 +0x9e5
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build/cluster.(*Builder).runBuildForArtifact(0xc0006185b0, {0x3e28b30, 0xc000b78600}, {0x3df6c80, 0xc0005ee780}, 0xc001264820, {0xc000c64080, 0x73}, {0x0, {0x0, ...}})
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/cluster/cluster.go:88 +0x3d1
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build/cluster.(*Builder).buildArtifact(0xc0006185b0, {0x3e28b30?, 0xc000b78600?}, {0x3df6c80, 0xc0005ee780}, 0xc001264820, {0xc000c64080, 0x73}, {0x0, {0x0, ...}})
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/cluster/cluster.go:71 +0x146
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build.(*BuilderMux).Build.func1({0x3e28b30, 0xc000b78600}, {0x3df6c80, 0xc0005ee780}, 0xc001264820, {0xc000c64080, 0x73}, {0x0, {0x0, 0x0, ...}})
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/builder_mux.go:112 +0x61d
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build.performBuild({0x3e28b30, 0xc000b78600}, {0x3df6c80, 0xc0005ee780}, 0x11?, {0xc0007a7d80?}, 0xc001264820, 0xc001d94b40)
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/scheduler.go:170 +0x213
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build.(*scheduler).build(0xc000370d90, {0x3e28b68, 0xc0005ee730}, 0xc001209080, {0xc0003ec738?}, 0x0)
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/scheduler.go:121 +0x70b
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/build.(*scheduler).run.func1()
	/tmpfs/src/git/skaffold-mirror/pkg/skaffold/build/scheduler.go:77 +0x2b
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/tmpfs/src/git/skaffold-mirror/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
	/tmpfs/src/git/skaffold-mirror/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96

Information

  • Skaffold version: v2.13.1
  • Operating system: Alpine Linux v3.18
  • Installed via: https://storage.googleapis.com/skaffold/releases/v2.13.1/skaffold-linux-amd64
---
apiVersion: skaffold/v2beta20
kind: Config
metadata:
  name: internal-admin-tools
build:
  tagPolicy:
    gitCommit:
      variant: CommitSha
  cluster:
    namespace: gitlab
    volumes:
      - name: kaniko-secret
        secret:
          secretName: kaniko-secret
  artifacts:
    - context: .
      image: us-docker.pkg.dev/recharge-infra-services-8b42/gcr.io/internal-admin-tools
      kaniko:
        verbosity: debug
        dockerfile: Dockerfile
        skipUnusedStages: true
        useNewRun: true
        env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /kaniko-secret/kaniko-secret.json
        cache:
          ttl: 336h
        volumeMounts:
          - name: kaniko-secret
            mountpath: /kaniko-secret
            readonly: true
        registryMirror: mirror.gcr.io
deploy:
  kubeContext: <redacted>
  statusCheckDeadlineSeconds: 300
  helm:
    flags:
      upgrade: [--timeout, 300s]
      install: [--timeout, 300s]
    releases:
      - name: internal-admin-tools
        chartPath: deploy/charts/internal-admin-tools
        namespace: default
        createNamespace: true
        upgradeOnChange: true
        wait: true
        valuesFiles:
          - deploy/local/values.yml
        artifactOverrides:
          image: us-docker.pkg.dev/recharge-infra-services-8b42/gcr.io/internal-admin-tools
profiles:
  - name: stage
    patches:
      - op: replace
        path: /deploy/helm/releases/0/namespace
        value: internal-admin-tools
      - op: replace
        path: /deploy/helm/releases/0/valuesFiles/0
        value: deploy/stage/values.yml
  - name: prod
    patches:
      - op: replace
        path: /deploy/helm/releases/0/namespace
        value: internal-admin-tools
      - op: replace
        path: /deploy/helm/releases/0/valuesFiles/0
        value: deploy/prod/values.yml

Steps to reproduce the behavior

Issue is very sporadic, running skaffold build many times will yield this result occasionally.

mjhuber avatar Aug 27 '24 20:08 mjhuber