Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-2118
-
Untriaged
-
0
-
Unknown
Description
Seen on both 108 and 109 -
On 109
2024-02-11T08:01:56.665-08:00 [Error] Indexer::handleUpdateIndexRState Unable to find Index 2162545866381380340 |
2024-02-11T08:01:56.665-08:00 [Info] Indexer::handleMergePartition Source 12271221548839648418 Target 4054995742004854515 |
panic: Inconsistent Internal State
|
|
goroutine 2048870 [running]: |
github.com/couchbase/indexing/secondary/common.CrashOnError(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509 |
github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).destTokenToMergeOrReady(0xc006740000, 0x1e02e8020026caf4, 0x13e25e0?, {0xc024bd8914?, 0xc04fa224e0?}, 0x1000000000019?, 0xc046aab7a0?) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2673 +0x28e |
github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).updateRStateToActive.func1(0x2) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2757 +0x225 |
created by github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).updateRStateToActive in goroutine 2048449 |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2753 +0x154 |
2024-02-11T08:01:58.042-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=1c901a3fe4a3d37e82947ba9a4a06a76 -isEnterprise=true --deploymentModel=default] |
One caveat is that the cluster has fewer index nodes and thus it might be undersized. At some point of the test, the cluster has ended up with 3 index nodes as opposed to 5 or 6. I'n unsure if that has anything to do with the panic. This was seen an hour or so after the rebalance failure reported in https://issues.couchbase.com/browse/MB-60758
cbcollect ->
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.67.zip
rl : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.67.zip
Older logs (n-1) ->
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.67.zip
Older logs (n-2)
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.67.zip
Older logs (n-3)
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.67.zip
Older logs (n-4)
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.67.zip