Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.6.0
-
7.6.0-1878
-
Untriaged
-
0
-
Unknown
Description
There were a few rebalance failures because of an indexer panic
Rebalance failure -
|
[user:error,2023-12-01T13:11:31.462-08:00,ns_1@172.23.97.67:<0.22199.0>:ns_orchestrator:log_rebalance_completion:1660]Rebalance exited with reason {service_rebalance_failed,index, |
{agent_died,<34800.16816.102>, |
{lost_connection,
|
{'ns_1@172.23.97.109',shutdown}}}}. |
Panic in 109
2023-12-01T13:11:30.543-08:00 [Fatal] PlasmaSlice::Statistics Error observed when processing Statistics on instId: 12823551696995497998, partnId: 2 |
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
|
panic: runtime error: invalid memory address or nil pointer dereference
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11dfbb4] |
|
goroutine 10729193 [running]: |
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics.func1()
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2849 +0xd4 |
panic({0x17c0cc0?, 0x2e28160?}) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/panic.go:914 +0x21f |
github.com/couchbase/nitro/skiplist.(*Skiplist).GetStats(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/stats.go:139 |
github.com/couchbase/plasma.(*Plasma).GetNumPages(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1162 |
github.com/couchbase/plasma.(*Plasma).getStats(_, _)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1175 +0xf4 |
github.com/couchbase/plasma.(*Plasma).GetPreparedStats(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1157 |
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics(0xc0acb4d800, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2864 +0x11c |
github.com/couchbase/indexing/secondary/indexer.(*storageMgr).getIndexStorageStats(0xc0055015f0, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1835 +0xbde |
github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats.func1()
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1588 +0x105 |
created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats in goroutine 328 |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1582 +0xb6 |
cbcollect ->
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701466219/collectinfo-2023-12-01T213855-ns_1%40172.23.97.67.zip
Similar panic was observed at a later stage on 108 and 171 nodes -
108
2023-12-01T15:51:32.790-08:00 [Fatal] PlasmaSlice::Statistics Error observed when processing Statistics on instId: 11751580713386540028, partnId: 2 |
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
|
panic: runtime error: invalid memory address or nil pointer dereference
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11dfbb4] |
|
goroutine 2587405 [running]: |
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics.func1()
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2849 +0xd4 |
panic({0x17c0cc0?, 0x2e28160?}) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/panic.go:914 +0x21f |
github.com/couchbase/nitro/skiplist.(*Skiplist).GetStats(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/stats.go:139 |
github.com/couchbase/plasma.(*Plasma).GetNumPages(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1162 |
github.com/couchbase/plasma.(*Plasma).getStats(_, _)
|
171
2023-12-01T16:31:48.399-08:00 [Info] LSS /data/@2i/bucket2_idx10_hnBUcvp5_17244616926946713535_0.index/docIndex(shard4425284426728282793) : LSS Snapshot SBHead :407809839 SBTail :447234048 |
|
2023-12-01T16:31:48.400-08:00 [Info] plasma node2node: initiating node transfer dst:https://172.23.97.109:9104 path:https://172.23.97.109:9104//plasma_storage_v1/24795aebc4a252aaed9055038d13b583_ShardTokenc7_8b_e0_73_ef_fd_d7_b4/4425284426728282793/bucket2_idx10_hnBUcvp5_17244616926946713535_0.index/docIndex/header.data |
2023-12-01T16:31:48.400-08:00 [Info] rpcClient(v1:0): rpc request:url:https://172.23.97.109:9104/rpc/OpenFile args:fInfo:{rPath:"/plasma_storage_v1/24795aebc4a252aaed9055038d13b583_ShardTokenc7_8b_e0_73_ef_fd_d7_b4/4425284426728282793/bucket2_idx10_hnBUcvp5_17244616926946713535_0.index/docIndex/header.data"} |
|
2023-12-01T16:31:48.405-08:00 [Info] rpcClient(v1:0): rpc request:url:https://172.23.97.109:9104/rpc/SyncAndCloseFile args:fh:{id:"/plasma_storage_v1/24795aebc4a252aaed9055038d13b583_ShardTokenc7_8b_e0_73_ef_fd_d7_b4/215531193164461311/shards/shard215531193164461311/data/log.00000000000000.data" gen:1701477106} |
|
2023-12-01T16:31:48.407-08:00 [Fatal] PlasmaSlice::Statistics Error observed when processing Statistics on instId: 8177042363828104497, partnId: 4 |
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
|
panic: runtime error: invalid memory address or nil pointer dereference
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11dfbb4] |
|
goroutine 293142565 [running]: |
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics.func1()
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2849 +0xd4 |
panic({0x17c0cc0?, 0x2e28160?}) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/panic.go:914 +0x21f |
github.com/couchbase/nitro/skiplist.(*Skiplist).GetStats(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/stats.go:139 |
github.com/couchbase/plasma.(*Plasma).GetNumPages(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1162 |
github.com/couchbase/plasma.(*Plasma).getStats(_, _)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1175 +0xf4 |
github.com/couchbase/plasma.(*Plasma).GetPreparedStats(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1157 |
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics(0xc017ae7200, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2864 +0x11c |
github.com/couchbase/indexing/secondary/indexer.(*storageMgr).getIndexStorageStats(0xc00489b380, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1835 +0xbde |
github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats.func1()
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1588 +0x105 |
created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats in goroutine 275 |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1582 +0xb6 |
2023-12-01T16:31:50.615-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=cc50de59d42d343c9915a25bcad47ac9 -isEnterprise=true --deploymentModel=default] |
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1701478450/collectinfo-2023-12-02T010244-ns_1%40172.23.97.67.zip