Details
-
Bug
-
Resolution: Incomplete
-
Major
-
4.5.0
-
4.5.0-2151
-
Untriaged
-
Windows 64-bit
-
Unknown
Description
steps:
Install CB server 4.5.0-X on 8 nodes( 6 CPU cores)
Setup 3 nodes cluster as SRC cluster.(172.23.105.143, 172.23.105.94, 172.23.106.29)
Setup 3 nodes cluster as DEST cluster.(172.23.105.248, 172.23.107.96, 172.23.106.83)
At each cluster, create 4 buckets
AbRegNums (0.7GB), MsgsCalls (0.7GB), RevAB (4.5GB) and UserInfo (0.5GB)
1) load data onr SRC in 20 hours.
2) After running load in more than 0.8 day, started replication from cluster SRC to cluster DEST.
3) rebalance in 172.23.107.48 at cluster SRC
rebalance in 172.23.107.227 at cluster DEST
4) click failover, Graceful Fail Over(rebalance) for node 172.23.107.48 in cluster A, add back(Delta Recovery) and rebalance - PASS
5) click failover, Hard Fail Over for node 172.23.107.48 in cluster SRC, add back(Full Recovery) and rebalance
result:
Bucket "AbRegNums" loaded on node 'ns_1@172.23.105.143' in 842 seconds. ns_memcached 000 ns_1@172.23.105.143 4:20:07 AM Mon Apr 18, 2016
|
Bucket "RevAB" loaded on node 'ns_1@172.23.105.143' in 809 seconds. ns_memcached 000 ns_1@172.23.105.143 4:19:33 AM Mon Apr 18, 2016
|
Resetting rebalance status since it's not really running ns_janitor 000 ns_1@172.23.105.94 4:17:51 AM Mon Apr 18, 2016
|
Bucket "UserInfo" loaded on node 'ns_1@172.23.105.143' in 705 seconds. ns_memcached 000 ns_1@172.23.105.143 4:17:49 AM Mon Apr 18, 2016
|
Bucket "MsgsCalls" loaded on node 'ns_1@172.23.105.143' in 708 seconds. ns_memcached 000 ns_1@172.23.105.143 4:17:49 AM Mon Apr 18, 2016
|
Couchbase Server has started on web port 8091 on node 'ns_1@172.23.105.143'. Version: "4.5.0-2151-enterprise". menelaus_sup 001 ns_1@172.23.105.143 4:05:40 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.106.29' saw that node 'ns_1@172.23.105.143' came up. Tags: [] ns_node_disco 004 ns_1@172.23.106.29 4:05:39 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.107.48' saw that node 'ns_1@172.23.105.143' came up. Tags: [] ns_node_disco 004 ns_1@172.23.107.48 4:05:39 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.105.143' synchronized otp cookie wpfbrqwphnbbeost from cluster ns_cookie_manager 002 ns_1@172.23.105.143 4:05:39 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.105.94' saw that node 'ns_1@172.23.105.143' came up. Tags: [] ns_node_disco 004 ns_1@172.23.105.94 4:05:39 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.106.29' saw that node 'ns_1@172.23.105.143' went down. Details: [{nodedown_reason,
|
connection_closed}] ns_node_disco 005 ns_1@172.23.106.29 4:02:57 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.107.48' saw that node 'ns_1@172.23.105.143' went down. Details: [{nodedown_reason,
|
connection_closed}] ns_node_disco 005 ns_1@172.23.107.48 4:02:56 AM Mon Apr 18, 2016
|
Node 'ns_1@172.23.105.94' saw that node 'ns_1@172.23.105.143' went down. Details: [{nodedown_reason,
|
connection_closed}] ns_node_disco 005 ns_1@172.23.105.94 4:02:55 AM Mon Apr 18, 2016
|
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master 000 ns_1@172.23.105.94 4:02:46 AM Mon Apr 18, 2016
|
Bucket "RevAB" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 3:50:37 AM Mon Apr 18, 2016
|
Bucket "RevAB" loaded on node 'ns_1@172.23.107.48' in 0 seconds. ns_memcached 000 ns_1@172.23.107.48 3:50:36 AM Mon Apr 18, 2016
|
Started rebalancing bucket RevAB ns_rebalancer 000 ns_1@172.23.105.143 3:50:35 AM Mon Apr 18, 2016
|
Bucket "MsgsCalls" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 3:44:22 AM Mon Apr 18, 2016
|
Bucket "MsgsCalls" loaded on node 'ns_1@172.23.107.48' in 0 seconds. ns_memcached 000 ns_1@172.23.107.48 3:44:20 AM Mon Apr 18, 2016
|
Started rebalancing bucket MsgsCalls ns_rebalancer 000 ns_1@172.23.105.143 3:44:19 AM Mon Apr 18, 2016
|
Bucket "AbRegNums" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 3:38:01 AM Mon Apr 18, 2016
|
Bucket "AbRegNums" loaded on node 'ns_1@172.23.107.48' in 0 seconds. ns_memcached 000 ns_1@172.23.107.48 3:37:59 AM Mon Apr 18, 2016
|
Started rebalancing bucket AbRegNums ns_rebalancer 000 ns_1@172.23.105.143 3:37:59 AM Mon Apr 18, 2016
|
Deleting old data files of bucket "AbRegNums" ns_storage_conf 000 ns_1@172.23.107.48 3:37:54 AM Mon Apr 18, 2016
|
Deleting old data files of bucket "UserInfo" ns_storage_conf 000 ns_1@172.23.107.48 3:37:54 AM Mon Apr 18, 2016
|
Deleting old data files of bucket "RevAB" ns_storage_conf 000 ns_1@172.23.107.48 3:37:54 AM Mon Apr 18, 2016
|
Deleting old data files of bucket "MsgsCalls" ns_storage_conf 000 ns_1@172.23.107.48 3:37:54 AM Mon Apr 18, 2016
|
Shutting down bucket "AbRegNums" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 3:36:56 AM Mon Apr 18, 2016
|
Shutting down bucket "MsgsCalls" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 3:36:55 AM Mon Apr 18, 2016
|
Shutting down bucket "RevAB" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 3:36:39 AM Mon Apr 18, 2016
|
Starting rebalance, KeepNodes = ['ns_1@172.23.105.143','ns_1@172.23.105.94',
|
'ns_1@172.23.106.29','ns_1@172.23.107.48'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
|
ns_orchestrator 004 ns_1@172.23.105.143 3:36:33 AM Mon Apr 18, 2016
|
Shutting down bucket "UserInfo" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 3:36:29 AM Mon Apr 18, 2016
|
Failed over 'ns_1@172.23.107.48': ok ns_rebalancer 000 ns_1@172.23.105.143 3:36:29 AM Mon Apr 18, 2016
|
Starting failing over 'ns_1@172.23.107.48' ns_rebalancer 000 ns_1@172.23.105.143 3:36:27 AM Mon Apr 18, 2016
|
Reset auto-failover count auto_failover 000 ns_1@172.23.105.143 2:32:50 AM Mon Apr 18, 2016
|
Rebalance completed successfully.
|
ns_orchestrator 001 ns_1@172.23.105.143 2:32:50 AM Mon Apr 18, 2016
|
Bucket "AbRegNums" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 2:25:12 AM Mon Apr 18, 2016
|
Started rebalancing bucket AbRegNums ns_rebalancer 000 ns_1@172.23.105.143 2:25:08 AM Mon Apr 18, 2016
|
Bucket "MsgsCalls" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 2:20:59 AM Mon Apr 18, 2016
|
Started rebalancing bucket MsgsCalls ns_rebalancer 000 ns_1@172.23.105.143 2:20:53 AM Mon Apr 18, 2016
|
Bucket "RevAB" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 2:17:51 AM Mon Apr 18, 2016
|
Started rebalancing bucket RevAB ns_rebalancer 000 ns_1@172.23.105.143 2:17:49 AM Mon Apr 18, 2016
|
Bucket "UserInfo" rebalance does not seem to be swap rebalance ns_vbucket_mover 000 ns_1@172.23.105.143 2:16:33 AM Mon Apr 18, 2016
|
Started rebalancing bucket UserInfo ns_rebalancer 000 ns_1@172.23.105.143 2:16:32 AM Mon Apr 18, 2016
|
Bucket "RevAB" loaded on node 'ns_1@172.23.107.48' in 100 seconds. ns_memcached 000 ns_1@172.23.107.48 2:16:31 AM Mon Apr 18, 2016
|
Bucket "UserInfo" loaded on node 'ns_1@172.23.107.48' in 99 seconds. ns_memcached 000 ns_1@172.23.107.48 2:16:30 AM Mon Apr 18, 2016
|
Bucket "MsgsCalls" loaded on node 'ns_1@172.23.107.48' in 49 seconds. ns_memcached 000 ns_1@172.23.107.48 2:15:40 AM Mon Apr 18, 2016
|
Bucket "AbRegNums" loaded on node 'ns_1@172.23.107.48' in 48 seconds. ns_memcached 000 ns_1@172.23.107.48 2:15:38 AM Mon Apr 18, 2016
|
Starting rebalance, KeepNodes = ['ns_1@172.23.105.143','ns_1@172.23.105.94',
|
'ns_1@172.23.106.29','ns_1@172.23.107.48'], EjectNodes = [], Failed over and being ejected nodes = [], Delta recovery nodes = ['ns_1@172.23.107.48'], Delta recovery buckets = all ns_orchestrator 004 ns_1@172.23.105.143 2:14:48 AM Mon Apr 18, 2016
|
Shutting down bucket "MsgsCalls" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 2:12:57 AM Mon Apr 18, 2016
|
Shutting down bucket "AbRegNums" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 2:12:53 AM Mon Apr 18, 2016
|
Failed over 'ns_1@172.23.107.48': ok ns_rebalancer 000 ns_1@172.23.105.143 2:12:33 AM Mon Apr 18, 2016
|
Shutting down bucket "RevAB" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 2:12:29 AM Mon Apr 18, 2016
|
Shutting down bucket "UserInfo" on 'ns_1@172.23.107.48' for deletion ns_memcached 000 ns_1@172.23.107.48 2:12:26 AM Mon Apr 18, 2016
|
Starting failing over 'ns_1@172.23.107.48' ns_rebalancer 000 ns_1@172.23.105.143 2:12:23 AM Mon Apr 18, 2016
|