cockroach opt: v23.1.14: partial index predicate does not exist in table metadata

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4954614362/?referrer=webhooks_plugin

Panic Message:

table_meta.go:443: partial index predicate does not exist in table metadata
(1) assertion failure
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/opt.(*TableMeta).PartialIndexPredicate
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/table_meta.go:443
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*scanIndexIter).ForEachStartingAfter
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:246
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*scanIndexIter).ForEach
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:208
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*CustomFuncs).GenerateIndexScans
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_funcs.go:51
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreScan
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:159
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreGroupMember
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:22
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/explorer.go:185
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:536
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeExpr
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:299
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:578
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:523
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).Optimize
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:269
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:598
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:245
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2060
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1567
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1037
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3119
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2183
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2188
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2105
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:902
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	src/runtime/asm_arm64.s:1172
Wraps: (3) partial index predicate does not exist in table metadata
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
-- report composition:
*errutil.leafError: partial index predicate does not exist in table metadata
table_meta.go:443: *withstack.withStack (top exception)
*assert.withAssertionFailure

Stacktrace (expand for inline code snippets):

src/runtime/asm_arm64.s in runtime.goexit at line 1172
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 902
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2105
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2188
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2183
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3119
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1037
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1567
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 2060
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 245
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 598
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).Optimize at line 269
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroup at line 523
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember at line 578
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeExpr at line 299
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroup at line 536
pkg/sql/opt/xform/explorer.go in pkg/sql/opt/xform.(*explorer).exploreGroup at line 185
bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go in pkg/sql/opt/xform.(*explorer).exploreGroupMember at line 22
bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go in pkg/sql/opt/xform.(*explorer).exploreScan at line 159
pkg/sql/opt/xform/scan_funcs.go in pkg/sql/opt/xform.(*CustomFuncs).GenerateIndexScans at line 51
pkg/sql/opt/xform/scan_index_iter.go in pkg/sql/opt/xform.(*scanIndexIter).ForEach at line 208
pkg/sql/opt/xform/scan_index_iter.go in pkg/sql/opt/xform.(*scanIndexIter).ForEachStartingAfter at line 246
pkg/sql/opt/table_meta.go in pkg/sql/opt.(*TableMeta).PartialIndexPredicate at line 443

Tags

Tag	Value
Command	start-single-node
Environment	v23.1.14
Go Version	go1.19.13
Platform	linux arm64
Distribution	CCL
Cockroach Release	v23.1.14
Cockroach SHA	b95f2225b09cf25f9f2be31ef6a6f65c73a1b081
# of CPUs	10
# of Goroutines	407

Jira issue: CRDB-35957

Feb 07 '24 19:02 cockroach-sentry

Hi, @yuzefovich Are you investigating this issue? I would like to debug it if no one is working on it.

Jun 28 '24 16:06 XiaochenCui

@XiaochenCui thanks for your interest! I don't think anyone is actively looking into this, so please feel free to take a stab.

Note, though, the automatically-filed sentry issues don't have a reproduction, so debugging this type of issue usually involves some trial and error in hopes of getting a repro while reading the relevant code to get ideas on what could have gone wrong. (And we often don't find anything, so the issue is closed as unactionable. Still, we try to add some more debugging information if possible to the code, like a better error message.)

Jun 28 '24 16:06 yuzefovich

The lack of reproduction doesn't matter, it's normal in db domain.

The only issue is I'm new to crdb and may give my feedback for this issue in a long time (like 2 months). Do you think it's ok? And I don't have the permission for sentry.io

Jun 28 '24 16:06 XiaochenCui

Yeah, totally, there is definitely no rush in looking into this (as you can see, this issue has stayed in the "triage" column for over 2 months at this point).

Re: access to sentry.io: I don't think that we'll be able to share access to sentry.io with you, but also the sentry event doesn't really contain any more debugging information other than what's included on this github issue (i.e. we only have a stack trace and the corresponding CRDB version). (I looked at the event for #124240 since the sentry event for #118913 seems to have been deleted.)

Jun 29 '24 16:06 yuzefovich

@yuzefovich Thanks for your information

Jun 29 '24 19:06 XiaochenCui

~~This is a canned plan - this logic is referenced in the stack trace:~~ Nevermind, I read it wrong. Will have to keep looking..

Aug 27 '24 21:08 DrewKimball

@XiaochenCui are you still working on this issue?

Sep 03 '24 18:09 DrewKimball

@XiaochenCui are you still working on this issue?

Not now, I did some research and failed to reproduce it in dev environment.

Sep 03 '24 20:09 XiaochenCui

Let's close this due to a lack of reproduction. We can investigate further if we see it again.

Sep 03 '24 20:09 DrewKimball

Let's close this due to a lack of reproduction. We can investigate further if we see it again.

Yeah, maybe someone who is really familiar with this part of code (like the author) can help with it.

Sep 03 '24 20:09 XiaochenCui