Details
Description
system tests:
2 cluster with 4 bucket, uni xdc replication
source:
172.23.105.158
172.23.105.156
172.23.105.157
172.23.105.22
destination:
172.23.105.160
172.23.105.159
172.23.105.206
172.23.105.207
1) remove node 172.23.105.159 on dest
2) remove node 172.23.105.22 on src
3) add 172.23.105.159 and 172.23.105.22 on src
results:
src cluster started replicate on itself
AbRegNums Version 2 this cluster bucket "AbRegNums" on cluster "172.23.105.159" Replicating on change
Delete
Settings
MsgsCalls Version 2 this cluster bucket "MsgsCalls" on cluster "172.23.105.159" Replicating on change
Delete
Settings
RevAB Version 2 this cluster bucket "RevAB" on cluster "172.23.105.159" Replicating on change
Delete
Settings
UserInfo Version 2 this cluster bucket "UserInfo" on cluster "172.23.105.159" Replicating on change
logs on destination when node 172.23.105.159 ejected:
Starting rebalance, KeepNodes = ['ns_1@172.23.105.160','ns_1@172.23.105.206',
'ns_1@172.23.105.207'], EjectNodes = ['ns_1@172.23.105.159'], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@172.23.105.159 08:20:13 - Fri May 2, 2014
logs on src when node 172.23.105.159 & 172.23.105.22 added:
Starting rebalance, KeepNodes = ['ns_1@172.23.105.156','ns_1@172.23.105.157',
'ns_1@172.23.105.158','ns_1@172.23.105.22',
'ns_1@172.23.105.159'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@172.23.105.156 09:30:06 - Fri May 2, 2014
Starting rebalance, KeepNodes = ['ns_1@172.23.105.156','ns_1@172.23.105.157',
'ns_1@172.23.105.158'], EjectNodes = ['ns_1@172.23.105.22'], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@172.23.105.156 08:20:11 - Fri May 2, 2014
also on this cluster I see a lot discomfiture:
1) stop all loaders but still see ~6K set(update) ops for bucket RevAB
2)MB-11032 DISK QUEUES for Active & resident items are not numbers, too big and does not correspond to total
3) rebalance stuck(will fill separate ticket)
4)
will keep the clusters alive:
source: http://172.23.105.156:8091/
destination:
http://172.23.105.160:8091/
https://s3.amazonaws.com/bugdb/jira/MB-11032/371fc18e/172.23.105.156-532014-621-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11032/371fc18e/172.23.105.157-532014-612-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11032/371fc18e/172.23.105.158-532014-62-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11032/371fc18e/172.23.105.22-532014-631-diag.zip