opt: v23.1.16: index out of bounds in SHOW RANGES

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5386176012/?referrer=webhooks_plugin

Panic Message:

catch.go:29: runtime error: index out of range [4] with length 4
(1) assertion failure
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/util/errorutil.ShouldCatch
  | 	github.com/cockroachdb/cockroach/pkg/util/errorutil/catch.go:29
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:217
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | runtime.goPanicIndex
  | 	GOROOT/src/runtime/panic.go:113
  | github.com/cockroachdb/cockroach/pkg/sql.(*optTable).Index
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:1221
  | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveIndex
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:318
  | github.com/cockroachdb/cockroach/pkg/sql/opt/cat.ResolveTableIndex
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/cat/utils.go:55
  | github.com/cockroachdb/cockroach/pkg/sql/delegate.(*delegator).delegateShowRanges
  | 	github.com/cockroachdb/cockroach/pkg/sql/delegate/show_ranges.go:95
  | github.com/cockroachdb/cockroach/pkg/sql/delegate.TryDelegate
  | 	github.com/cockroachdb/cockroach/pkg/sql/delegate/delegate.go:114
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:426
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:277
  | github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:251
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:584
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:245
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2060
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1567
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1037
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3119
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2181
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2186
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2103
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:902
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (3) runtime error: index out of range [4] with length 4
  | -- cause hidden behind barrier
  | runtime error: index out of range [4] with length 4
  | (1) runtime error: index out of range [4] with length 4
  | Error types: (1) runtime.boundsError
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: masked error: runtime error: index out of range [4] with length 4
catch.go:29: *withstack.withStack (top exception)
*assert.withAssertionFailure

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in 37f7574

	reservedOwned = false // We're about to pass ownership away.
	retErr = sqlServer.ServeConn(
	ctx,

cockroach/pkg/sql/conn_executor.go

Lines 901 to 903 in 37f7574

	}(ctx, h)
	return h.ex.run(ctx, s.pool, reserved, cancel)
	}

cockroach/pkg/sql/conn_executor.go

Lines 2102 to 2104 in 37f7574

	var err error
	if err = ex.execCmd(); err != nil {
	// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2185 to 2187 in 37f7574

	return err
	}()
	// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2180 to 2182 in 37f7574

	tcmd.LastInBatch || !implicitTxnForBatch)
	ev, payload, err = ex.execStmt(
	ctx, tcmd.Statement, nil /* portal */, nil /* pinfo */, stmtRes, canAutoCommit,

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 37f7574

	}
	err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3118 to 3120 in 37f7574

	} else {
	err = op(ctx)
	}

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 37f7574

	err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
	return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1036 to 1038 in 37f7574

	if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
	stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1566 to 1568 in 37f7574

	// between here and there needs to happen even if there's an error.
	err = ex.makeExecPlan(ctx, planner)
	defer planner.curPlan.close(ctx)

cockroach/pkg/sql/conn_executor_exec.go

Lines 2059 to 2061 in 37f7574

	func (ex *connExecutor) makeExecPlan(ctx context.Context, planner *planner) error {
	if err := planner.makeOptimizerPlan(ctx); err != nil {
	log.VEventf(ctx, 1, "optimizer plan failed: %v", err)

cockroach/pkg/sql/plan_opt.go

Lines 244 to 246 in 37f7574

	execMemo, err := opc.buildExecMemo(ctx)
	if err != nil {

cockroach/pkg/sql/plan_opt.go

Lines 583 to 585 in 37f7574

	bld := optbuilder.New(ctx, &p.semaCtx, p.EvalContext(), opc.catalog, f, opc.p.stmt.AST)
	if err := bld.Build(); err != nil {
	return nil, err

cockroach/pkg/sql/opt/optbuilder/builder.go

Lines 250 to 252 in 37f7574

	// and physical properties.
	outScope := b.buildStmtAtRoot(b.stmt, nil /* desiredTypes */)

cockroach/pkg/sql/opt/optbuilder/builder.go

Lines 276 to 278 in 37f7574

	b.ctes = nil
	outScope = b.buildStmt(stmt, desiredTypes, inScope)
	// Build With operators for any CTEs hoisted to the top level.

cockroach/pkg/sql/opt/optbuilder/builder.go

Lines 425 to 427 in 37f7574

	// delegate functionality.
	newStmt, err := delegate.TryDelegate(
	b.ctx,

cockroach/pkg/sql/delegate/delegate.go

Lines 113 to 115 in 37f7574

	}
	return d.delegateShowRanges(t)

cockroach/pkg/sql/delegate/show_ranges.go

Lines 94 to 96 in 37f7574

	var err error
	idx, resName, err = cat.ResolveTableIndex(
	d.ctx, d.catalog, cat.Flags{

cockroach/pkg/sql/opt/cat/utils.go

Lines 54 to 56 in 37f7574

	) (Index, DataSourceName, error) {
	return catalog.ResolveIndex(ctx, flags, name)
	}

cockroach/pkg/sql/opt_catalog.go

Lines 317 to 319 in 37f7574

	return table.Index(idx.Ordinal()), oc.tn, nil
	}

cockroach/pkg/sql/opt_catalog.go

Lines 1220 to 1222 in 37f7574

	func (ot *optTable) Index(i cat.IndexOrdinal) cat.Index {
	return &ot.indexes[i]
	}

GOROOT/src/runtime/panic.go#L112-L114
GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/opt/optbuilder/builder.go

Lines 216 to 218 in 37f7574

	// manipulate locks.
	if ok, e := errorutil.ShouldCatch(r); ok {
	err = e

cockroach/pkg/util/errorutil/catch.go

Lines 28 to 30 in 37f7574

	// get reported to Sentry.
	err = errors.HandleAsAssertionFailure(err)
	}

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 902
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2103
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2186
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2181
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3119
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1037
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1567
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 2060
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 245
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 584
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).Build at line 251
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot at line 277
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmt at line 426
pkg/sql/delegate/delegate.go in pkg/sql/delegate.TryDelegate at line 114
pkg/sql/delegate/show_ranges.go in pkg/sql/delegate.(*delegator).delegateShowRanges at line 95
pkg/sql/opt/cat/utils.go in pkg/sql/opt/cat.ResolveTableIndex at line 55
pkg/sql/opt_catalog.go in pkg/sql.(*optCatalog).ResolveIndex at line 318
pkg/sql/opt_catalog.go in pkg/sql.(*optTable).Index at line 1221
GOROOT/src/runtime/panic.go in runtime.goPanicIndex at line 113
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).Build.func1 at line 217
pkg/util/errorutil/catch.go in pkg/util/errorutil.ShouldCatch at line 29

Tags

Tag	Value
Command	server
Environment	v23.1.16
Go Version	go1.19.13
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.16
Cockroach SHA	37f7574
# of CPUs	64
# of Goroutines	6050

Jira issue: CRDB-38933

[yuzefovich]

Hm, I tried a few things but couldn't reproduce this (like indexes on virtual columns and indexes being backfilled while SHOW RANGES is executed). My guess is that the problem is very rare and is due to a mismatch between optTable.indexes and tabledesc.ordinal, perhaps with a concurrent schema change involved.