kv/kvserver: TestCrashWhileTruncatingSideloadedEntries failed

[cockroach-teamcity]

kv/kvserver.TestCrashWhileTruncatingSideloadedEntries failed on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1693
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestCrashWhileTruncatingSideloadedEntries
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_log_queue_test.go:354
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	Wraps: (4) L6: 000034: file 000034 (type 2) unknown to the objstorage provider: file does not exist
        	            	    └─ Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/pebble.checkConsistency
        	            	  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:1288
        	            	  | [...repeated from below...]
        	            	      └─ Wraps: (6) L6: 000034
        	            	        └─ Wraps: (7) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).Lookup
        	            	  | 	github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:399
        	            	  | github.com/cockroachdb/pebble.checkConsistency
        	            	  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:1279
        	            	  | github.com/cockroachdb/pebble.Open
        	            	  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/open.go:393
        	            	  | github.com/cockroachdb/cockroach/pkg/storage.newPebble
        	            	  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1327
        	            	  | github.com/cockroachdb/cockroach/pkg/storage.Open
        	            	  | 	github.com/cockroachdb/cockroach/pkg/storage/open.go:502
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*Config).CreateEngines
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/config.go:855
        	            	  | github.com/cockroachdb/cockroach/pkg/server.NewServer
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/server.go:297
        	            	  | github.com/cockroachdb/cockroach/pkg/server.testServerFactoryImpl.New
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/testserver.go:2455
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.NewServer
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:327
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1749
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1693
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestCrashWhileTruncatingSideloadedEntries
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_log_queue_test.go:354
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	          └─ Wraps: (8) file 000034 (type 2) unknown to the objstorage provider
        	            	            └─ Wraps: (9) file does not exist
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.withPrefix (9) *errors.errorString
        	Test:       	TestCrashWhileTruncatingSideloadedEntries
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestCrashWhileTruncatingSideloadedEntries2390546897
--- FAIL: TestCrashWhileTruncatingSideloadedEntries (0.83s)

Parameters:

• attempt=1
• run=8
• shard=34

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-38913

[kvoli]

Fails setting up Pebble storage engine upon restart, complaining about a file not being known to the object storage provider 000034.

The file doesn't show up in the local FS, perhaps because it is mean to be remote?

Details

    client_raft_log_queue_test.go:353: FS after restart:
                  /
             830    000002.log
            1232    000004.sst
             990    000006.sst
             966    000007.sst
            1067    000008.sst
              52    000010.log
           10884    000011.log
            1668    000013.sst
             914    000014.sst
             914    000015.sst
             914    000016.sst
             914    000017.sst
             914    000018.sst
             914    000019.sst
             914    000020.sst
             914    000021.sst
             914    000022.sst
             914    000023.sst
             914    000024.sst
             914    000025.sst
             914    000026.sst
             914    000027.sst
             914    000028.sst
             914    000029.sst
             914    000030.sst
             914    000031.sst
             914    000032.sst
             914    000033.sst
               0    LOCK
            2148    MANIFEST-000001
            2590    OPTIONS-000003
              16    REMOTE-OBJ-CATALOG-000001
               8    STORAGE_MIN_VERSION
                    auxiliary/
                      sideloading/
                        r0XXXX/
                          r69/
             914            i41.t6
             914            i42.t6
                      sstsnapshot/
               0    marker.format-version.000005.017
               0    marker.manifest.000001.MANIFEST-000001
               0    marker.remote-obj-catalog.000001.REMOTE-OBJ-CATALOG-000001

Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_log_queue_test.go:354
Error:      	Received unexpected error:
            	failed to create engines: L6: 000034: file 000034 (type 2) unknown to the objstorage provider: file does not exist

This is potentially concerning? cc @pav-kv when you have a chance could you take a quick look.

[pav-kv]

This test emulates a crash by effectively switching off disk flushes/fsyncs. Possibly a bug in the test or the in-memory test-only storage that implements this switch-off.