[System Test] Indexer panic leading to rebalance failure - error observed when processing Statistics

2024-01-26T07:34:20.639-08:00 [Info] LSS /data/@2i/shards/shard6171059609295495146/data(shard6171059609295495146) : LSSCleaner.Stop: Begin stopping data LSS Cleaner

2024-01-26T07:34:20.639-08:00 [Info] LSS /data/@2i/shards/shard6171059609295495146/data(shard6171059609295495146) : logCleaner: starting... frag 31, data: 173249462, used: 251198611 log:(2577576261 - 2828787712) head:2577576261

2024-01-26T07:34:20.639-08:00 [Info] LSS /data/@2i/shards/shard6171059609295495146/data(shard6171059609295495146) : logCleaner: completed... frag 100, data: 0, used: 87829896, disk: 222734193, relocated: 248, rewrites: 249, retries: 0, skipped: 1793, cleaned:167819423, log:(2745395684 - 2834071552), disk:(2611337359-2834071552), activeFragRatio:30, activeMaxFragRatio:80, run:2 duration:3468 ms elapsed: 253307 ms

2024-01-26T07:34:20.719-08:00 [Fatal] PlasmaSlice::Statistics Error observed when processing Statistics on instId: 12822323619353246983, partnId: 2

panic: runtime error: invalid memory address or nil pointer dereference [recovered]

	panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11e3053]

goroutine 98706142 [running]:

github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics.func1()

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2883 +0xd4

panic({0x17cba80?, 0x2e31180?})

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/panic.go:914 +0x21f

github.com/couchbase/nitro/skiplist.(*Skiplist).GetStats(...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/stats.go:139

github.com/couchbase/plasma.(*Plasma).GetNumPages(...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1158

github.com/couchbase/plasma.(*Plasma).getStats(_, _)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1171 +0xf3

github.com/couchbase/plasma.(*Plasma).GetPreparedStats(...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1153

github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics(0xc064e60000, 0x0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2898 +0x11c

github.com/couchbase/indexing/secondary/indexer.(*storageMgr).getIndexStorageStats(0xc0063e89c0, 0x0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1849 +0xbde

github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats.func1()

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1602 +0x105

created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats in goroutine 36969

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1596 +0xb6

2024-01-26T07:34:23.077-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=5f885f5aca540d103279b49aecd32eeb -isEnterprise=true --deploymentModel=default]

2024-01-26T07:34:23.078-08:00 [Info] Indexer DeploymentModel is set to: default