Details
-
Bug
-
Resolution: Done
-
Major
-
Cheshire-Cat
-
6.5.0-6299 -> 7.0.0-4291
-
Untriaged
-
1
-
Unknown
Description
I have not gotten a local repro, but on jenkins this job is semi-regularly failing due to panics.
This test has the following steps:
3 nodes
1 node (kv,n1ql) - 172.23.123.64
2 node(index) - 172.23.123.62 / 172.23.123.63
1 bucket - standard_bucket0
Before upgrade we create the following indexes:
CREATE INDEX index_name_0 ON standard_bucket0(job_title) USING GSI WITH
BUILD INDEX on standard_bucket0(index_name_0) USING GSI
CREATE INDEX idx_pending ON standard_bucket0(createdDateTime) WHERE status = 'PENDING'
CREATE INDEX idx_success ON standard_bucket0(createdDateTime) WHERE status = 'SUCCESS'
CREATE INDEX idx_error ON standard_bucket0(createdDateTime) WHERE status = 'ERROR'
Then the test populates the following data:
INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K1',
)
INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K2',
)
INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K3',
)
INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K4',
)
The test then updates the data:
UPDATE `standard_bucket0` USE KEYS ['K1', 'K2', 'K3'] SET status = 'SUCCESS'
UPDATE `standard_bucket0` USE KEYS 'K4' SET status = 'ERROR'
After this the test then runs the upgrade to 7.0.0 by stopping the couchbase service and performing upgrade then bringing the nodes back online. The order is .64 (kv,n1ql), .62(index), .63(index)
After upgrade is complete we see this error:
[2021-01-24 09:03:02,720] - [rest_client:1021] ERROR - socket error while connecting to http://172.23.123.64:8091/indexStatus error [Errno 99] Cannot assign requested address
And upon inspecting logs we see a panic on .63, will attach logs from all servers
2021-01-24T08:58:02.013-08:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
InstanceId: 12081943181962620740 Name: index_name_0 Keyspace: standard_bucket0:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x841947]
goroutine 1 [running]:
github.com/couchbase/indexing/secondary/common/collections.(*CollectionManifest).GetCollectionID(0x0, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0xc000105601, 0xc004c3e2c8)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/collections/collection_defs.go:24 +0x37
github.com/couchbase/indexing/secondary/dcp.(*Pool).GetCollectionID(...)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:861
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).GetCollectionID(0xc000310900, 0xc004c28460, 0x10, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0x1, 0x2)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:534 +0xb4
github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).ValidateCollectionID.func1(...)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:991
github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).ValidateCollectionID(0xc00034a240, 0xc004c28460, 0x10, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0x125816f, 0x1, 0x100)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:998 +0xf3
github.com/couchbase/indexing/secondary/indexer.(*indexer).validateIndexInstMap(0xc0012c0700)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7130 +0x575
github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1(0xc0012c0700, 0xc000100b40, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6130 +0x3fb
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc00013d710, 0x26, 0x7ffc000001ed, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:397 +0x1a71
main.main()
goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:169 +0x11c4
2021-01-24T08:58:14.934-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=cad378e00fba89ff0284fe3dd4f47ad1 -ipv6=false -isEnterprise=true]