Details
-
Bug
-
Resolution: Fixed
-
Major
-
3.1.5
-
4.5.0-2151
-
Untriaged
-
Windows 64-bit
-
Unknown
-
KV: April 13 - April 26
Description
Clone of MB-19255.
steps:
Install CB server 4.5.0-X on 8 nodes( 6 CPU cores)
Setup 3 nodes cluster as SRC cluster.(172.23.105.143, 172.23.105.94, 172.23.106.29)
Setup 3 nodes cluster as DEST cluster.(172.23.105.248, 172.23.107.96, 172.23.106.83)
At each cluster, create 4 buckets
AbRegNums (0.7GB), MsgsCalls (0.7GB), RevAB (4.5GB) and UserInfo (0.5GB)
1) load data onr SRC in 20 hours.
2) After running load in more than 0.8 day, started replication from cluster SRC to cluster DEST.
3) rebalance in 172.23.107.48 at cluster SRC
rebalance in 172.23.107.227 at cluster DEST
result: Rebalance exited with reason bad_replicas
Rebalance exited with reason bad_replicas
|
ns_orchestrator 002 ns_1@172.23.105.248 12:45:28 AM Mon Apr 18, 2016
|
Bad replicators after rebalance:
|
Missing = [{'ns_1@172.23.107.96','ns_1@172.23.105.248',683},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',684},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',685},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',686},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',687},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',688},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',689},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',690},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',691},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',692},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',693},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',694},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',695},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',696},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',697},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',698},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',699},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',700},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',701},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',702},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',703},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',704},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',705},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',706},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',707},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',708},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',709},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',710},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',711},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',712},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',713},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',714},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',715},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',716},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',717},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',718},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',719},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',720},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',721},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',722},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',723},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',724},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',725},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',726},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',727},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',728},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',729},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',730},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',731},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',732},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',733},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',734},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',735},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',736},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',737},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',738},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',739},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',740},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',741},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',742},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',743},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',744},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',745},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',746},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',747},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',748},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',749},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',750},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',751},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',752},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',753},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',754},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',755},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',756},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',757},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',758},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',759},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',760},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',761},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',762},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',763},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',764},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',765},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',766},
|
{'ns_1@172.23.107.96','ns_1@172.23.105.248',767}]
|
Extras = [] ns_rebalancer 002 ns_1@172.23.105.248 12:45:28 AM Mon Apr 18, 2016
|
Bucket "RevAB" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.248 12:40:35 AM Mon Apr 18, 2016
|
Bucket "RevAB" loaded on node 'ns_1@172.23.107.227' in 0 seconds. ns_memcached 000 ns_1@172.23.107.227 12:40:32 AM Mon Apr 18, 2016
|
Started rebalancing bucket RevAB ns_rebalancer 000 ns_1@172.23.105.248 12:40:32 AM Mon Apr 18, 2016
|
Bucket "UserInfo" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.248 12:34:40 AM Mon Apr 18, 2016
|
Bucket "UserInfo" loaded on node 'ns_1@172.23.107.227' in 0 seconds. ns_memcached 000 ns_1@172.23.107.227 12:34:36 AM Mon Apr 18, 2016
|
Started rebalancing bucket UserInfo ns_rebalancer 000 ns_1@172.23.105.248 12:34:35 AM Mon Apr 18, 2016
|
Starting rebalance, KeepNodes = ['ns_1@172.23.105.248','ns_1@172.23.106.83',
|
'ns_1@172.23.107.227','ns_1@172.23.107.96'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
|
ns_orchestrator 004 ns_1@172.23.105.248 12:34:33 AM Mon Apr 18, 2016
|
Failed to add node 172.23.107.227:8091 to cluster. Prepare join failed. Node is already part of cluster. ns_cluster 005 ns_1@172.23.105.248 12:33:06 AM Mon Apr 18, 2016
|
Node ns_1@172.23.107.227 joined cluster ns_cluster 003 ns_1@172.23.107.227 12:30:33 AM Mon Apr 18, 2016
|
Couchbase Server has started on web port 8091 on node 'ns_1@172.23.107.227'. Version: "4.5.0-2151-enterprise". menelaus_sup 001 ns_1@172.23.107.227 12:30:32 AM Mon Apr 18, 2016
|