Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Base build: 6.0.0-1693-enterprise
Target build: 7.0.0-4476-enterprise
Scenario:
- 4 node cluster (6.0.0-1693)
- Couchbase bucket with replica=1
- Load bucket until 70% DGM
- Start online swap_upgrade to build **7.0.0-4476 (one node at a time)
Observation:
Rebalance failed during the following swap-rebalance:
+----------------+----------+-----------------------+---------------+--------------+
|
| Nodes | Services | Version | CPU | Status |
|
+----------------+----------+-----------------------+---------------+--------------+
|
| 172.23.105.212 | kv | 6.0.0-1693-enterprise | 3.0303030303 | --- OUT ---> |
|
| 172.23.105.244 | kv | 7.0.0-4476-enterprise | 20.0251256281 | Cluster node |
|
| 172.23.105.155 | kv | 7.0.0-4476-enterprise | 24.6067985794 | Cluster node |
|
| 172.23.105.213 | kv | 6.0.0-1693-enterprise | 3.26633165829 | Cluster node |
|
| 172.23.105.211 | None | | | <--- IN --- |
|
+----------------+----------+-----------------------+---------------+--------------+
|
Rebalance failure logs,
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit, {'EXIT',<0.21110.3>,
|
{{wait_seqno_persisted_failed,"default",723, 26728,
|
[{'ns_1@172.23.105.211', {'EXIT',
|
{socket_closed, {gen_server,call,
|
[{'janitor_agent-default', 'ns_1@172.23.105.211'},
|
{if_rebalance,<0.8319.3>,
|
{wait_seqno_persisted,723,26728}}, infinity]}}}}]},
|
[{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,488}]},
|
{proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,234}]}]}}}}.
|
|
|
Rebalance Operation Id = f7f0922241e2777b2eefdde991763996
|
|
|
Worker <0.21022.3> (for action {move,{723,
|
['ns_1@172.23.105.212', 'ns_1@172.23.105.213'],
|
['ns_1@172.23.105.211', 'ns_1@172.23.105.213'],
|
[]}}) exited with reason {unexpected_exit, {'EXIT', <0.21110.3>,
|
{{wait_seqno_persisted_failed, "default", 723,26728,
|
[{'ns_1@172.23.105.211', {'EXIT',
|
{socket_closed, {gen_server, call,
|
[{'janitor_agent-default', 'ns_1@172.23.105.211'},
|
{if_rebalance, <0.8319.3>,
|
{wait_seqno_persisted, 723, 26728}}, infinity]}}}}]},
|
[{ns_single_vbucket_mover,
|
'-wait_seqno_persisted_many/5-fun-2-', 5, [{file, "src/ns_single_vbucket_mover.erl"}, {line, 488}]},
|
{proc_lib, init_p,3, [{file, "proc_lib.erl"}, {line, 234}]}]}}}
|
TAF test:
upgrade.durability_upgrade.UpgradeTests:
|
test_upgrade,nodes_init=4,replicas=1,update_nodes=kv,num_items=50000,upgrade_type=online_swap,initial_version=6.0.0-1693,upgrade_with_data_load=False,skip_buckets_handle=True,durability=MAJORITY,upgrade_version=7.0.0-4476,log_level=debug,active_resident_threshold=70
|