Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
Cheshire-Cat
-
6.6.2-9588 -> 7.0.0-5141
-
Untriaged
-
Centos 64-bit
-
1
-
Yes
Description
Scripts to Repro
1. Run the 6.6.2 longevity test for 3 days.
./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. It had 27 nodes at the end of the test.
3. Added 6 7.0.0(172.23.105.102,172.23.105.62,172.23.106.232,172.23.106.239,172.23.106.37, 172.23.106.246) nodes and rebalanced in and removed 6 node from 6.6.2(172.23.110.75,172.23.110.76,172.23.105.61,172.23.106.191,172.23.106.209,172.23.106.70)
and rebalanced out.
4. Failed over 6 nodes and graceful failover + recovery + rebalance.
5. Now swap rebalance 6 nodes. 2 data + 2 index + 1 eventing + 1 analytics.
6. Started a 5 node (2 indexing + 1 data + 1 eventing + 1 analytics) graceful failover + delta recovery + rebalance.
7. Upgraded rest of the nodes using offline upgrade.
Post upgrade errors like the following were seen in fts logs.
172.23.106.239 : fts
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-05-15T04:05:41.217-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: social, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-05-15T04:05:41.228-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: good_state, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:63
|
cbcollect_info attached. This was not seen in upgrade during 6.6.2->9588 to 7.0.0-5033.