Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
6.6.2-9588 ----> 7.0.0-4979
-
Untriaged
-
Centos 64-bit
-
1
-
No
Description
Steps to Repro
It is an essentially an upgrade of the system test cluster.
1. Start a 6.6.2 system test longevity run.
2. It has following cluster setup
- * 9 data nodes
- * 3 analytics nodes
- * 3 eventing nodes
- * 4 indexing nodes
- * 3 search nodes
- * 3 query nodes
3. It has 10 buckets, fts indexes, analytics datasets, 2i indexes, eventing functions.
4. We do a swap rebalance of 6 node(1 data, 1 index, 1 analytics, 1 fts, 1 query, 1 eventing) which has 6.6.2-9588 with 7.0.0-4979. This woks fine.
5. Failover one fts node 6.6.2-9588 - 172.23.106.207
6. Failover one n1ql node 6.6.2-9588 - 172.23.106.191
7. Now try to graceful failover one 6.6.2-9588 - 172.23.105.90
8. Now I hit into MB-45767.
9. To proceed with the upgrade of the cluster at this point I do multi node hard failover of the following nodes.
172.23.105.90
|
172.23.105.62
|
172.23.105.118
|
172.23.105.25
|
10. Run the following command on all the nodes (172.23.105.90,172.23.105.62,172.23.105.118,172.23.105.25,172.23.106.207,172.23.106.191).
systemctl stop couchbase-server
|
rpm -U http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4979/couchbase-server-enterprise-7.0.0-4979-centos7.x86_64.rpm
|
I left the cluster up for 8-10 hours in mixed mode cluster. Saw ns_serv exiting on the following 2 nodes.
172.23.106.191 :
[user:info,2021-04-19T19:27:54.001-07:00,ns_1@172.23.106.191:<0.604.0>:ns_log:crash_consumption_loop:63]Service 'ns_server' exited with status 137. Restarting. Messages:
|
172.23.106.225:
[user:info,2021-04-19T20:32:34.379-07:00,ns_1@172.23.106.225:<0.593.0>:ns_log:crash_consumption_loop:63]Service 'goxdcr' exited with status 2. Restarting. Messages:
|
[user:info,2021-04-19T20:32:34.382-07:00,ns_1@172.23.106.225:<0.593.0>:ns_log:crash_consumption_loop:63]Service 'ns_server' exited with status 137. Restarting. Messages:
|
cbcollect_info attached.
Attachments
Issue Links
- depends on
-
MB-46375 Build new package with erlang 22.3.4.15 and fix for MB-45793
- Closed