Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 6.5.1-6299 --> 7.0.0-4291
Panic seen in the below test with job cen7-2i-plasma-set5-job1-upgrade-6-5-1_RED
./testrunner -i /tmp/centos_upgrade2i_6_5_1_red.ini -p upgrade_to=7.0.0-4291,initial_version=6.5.1-6299,get-cbcollect-info=False,get-coredumps=False -t gsi.upgrade_gsi.UpgradeSecondaryIndex.test_online_upgrade_with_rebalance,nodes_init=3,services_init=kv:n1ql-index-index,standard_buckets=1,dataset=default,scan_consistency=request_plus,groups=simple,init_nodes=False,gsi_type=forestdb,nodes_out=2,nodes_out_dist=index:2,nodes_in=2,services_in=index:2,before=create_index,disable_plasma_upgrade=True
Steps in the test
- Setup 3 node 6.5.1-6299 cluster with kv+n1ql, index , index services
- create standard_bucket0 bucket and load 98784.0 docs
- create and build below indexes
* CREATE PRIMARY INDEX ON standard_bucket0 USING GSI* CREATE INDEX index_name_0 ON standard_bucket0(job_title) USING GSI WITH {'defer_build': True}* QUERY BUILD INDEX on standard_bucket0(index_name_0) USING GSI* QUERY CREATE INDEX employee1e2bf8807f254cc9ba1b61b874be5e7ajob_title ON standard_bucket0(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'nodes': ['172.23.105.15:8091'], 'defer_build': True}* QUERY CREATE INDEX employee1e2bf8807f254cc9ba1b61b874be5e7ajoin_yr ON standard_bucket0(join_yr) WHERE join_yr > 2010 and join_yr < 2014 USING GSI WITH {'nodes': ['172.23.105.18:8091'], 'defer_build': True}* QUERY BUILD INDEX on standard_bucket0(employee1e2bf8807f254cc9ba1b61b874be5e7ajob_title,employee1e2bf8807f254cc9ba1b61b874be5e7ajoin_yr) USING GSI
- Start mutations on standard_bucket0
- Parallely rebalance in index node with 7.0.0-4291
Rebalance failed and below Panic seen 7.0.0-4291 indexer logs. 172.23.105.21 is the indexer with panic
panic: runtime error: invalid memory address or nil pointer dereference
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x841947]
|
|
|
goroutine 176 [running]:
|
github.com/couchbase/indexing/secondary/common/collections.(*CollectionManifest).GetCollectionID(0x0, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0xc0069c8f01, 0xc0069c8f00)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/collections/collection_defs.go:24 +0x37
|
github.com/couchbase/indexing/secondary/dcp.(*Pool).GetCollectionID(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:861
|
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).GetCollectionID(0xc000282000, 0xc007352c0e, 0x10, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0x8, 0xe)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:534 +0xb4
|
github.com/couchbase/indexing/secondary/manager.(*IndexManager).getKeyspaceForCleanup(0xc00236e090, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:768 +0x250
|
github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0xc00236e090, 0xc004772ba0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:725 +0x12c
|
created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:213 +0xab4
|
2021-01-24T04:30:52.796-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=98257c32e8df65177c5232d6189302e3 -ipv6=false -isEnterprise=true]
|