panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod [0.23.5]

[gregakinman]

Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x1b1b5e0), concrete:(*runtime._type)(0x1c2ae20), asserted:(*runtime._type)(0x1ddf8c0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod)
goroutine 144 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1b8e7c0?, 0xc002ed6e40})
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00005f7d0?})
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x1b8e7c0, 0xc002ed6e40})
	/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).addEventHandlersPod.func3({0x1c2ae20?, 0xc002b3bfe0?})
	/clickhouse-operator/pkg/controller/chi/controller.go:431 +0x245
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
	/clickhouse-operator/vendor/k8s.io/client-go/tools/cache/controller.go:257
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/clickhouse-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:978 +0xaf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0003ec738?, {0x20d1e00, 0xc000430480}, 0x1, 0xc00015c000)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2020202020202020?, 0x3b9aca00, 0x0, 0x20?, 0x203a6e6f6d6d6f63?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000618870)
	/clickhouse-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x85
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod
goroutine 144 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00005f7d0?})
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x1b8e7c0, 0xc002ed6e40})
	/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).addEventHandlersPod.func3({0x1c2ae20?, 0xc002b3bfe0?})
	/clickhouse-operator/pkg/controller/chi/controller.go:431 +0x245
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
	/clickhouse-operator/vendor/k8s.io/client-go/tools/cache/controller.go:257
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/clickhouse-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:978 +0xaf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0003ec738?, {0x20d1e00, 0xc000430480}, 0x1, 0xc00015c000)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2020202020202020?, 0x3b9aca00, 0x0, 0x20?, 0x203a6e6f6d6d6f63?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000618870)
	/clickhouse-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:967 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x85

[Slach]

could you provide little bit more logs from clickouse-operator` before panic?

[alex-zaitsev]

@gregakinman , 0.23.5 is almost 2 years old, there could be incompatibilities with new k8s versions. Consider an upgrade