Details
-
Bug
-
Resolution: Fixed
-
Critical
-
3.0
-
Security Level: Public
-
None
-
Centos 6.4 64-bit
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
Two centos 6.4 64-bit
10.3.121.15
10.3.121.16
Install couchbase server 3.0.0-876
Create default bucket on node 10.3.121.15
Load 1K to default bucket
Add node 10.3.121.16 to node 10.3.121.15
Rebalance. Passed
Kill erlang in both node to do warmup test. Passed
Clean up. Remove node 10.3.121.16 out of cluster. Failed. Rebalance hang
Cluster is in hang state now.
Note:
This bug may relate to bug MB-11459
This warm up test passed in ubuntu 12.04 64-bit with build ubuntu_12_04_3.0.0-867 for openssl 1
This warm up test also passed in ubuntu 12.04 64-bit with build 3.0.0-876 for openssl 0.9.8
This warm up test passed in centos 6.4 64-bit with build centos6_3.0.0-858 for openssl 1 as show in log below
[2014-06-23 12:14:39,097] - [cluster_helper:253] INFO - rebalancing all nodes in order to remove nodes
[2014-06-23 12:14:39,104] - [rest_client:1087] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.182%2Cns_1%40172.23.105.210%2Cns_1%40172.23.105.132&user=Administrator&knownNodes=ns_1%40172.23.105.182%2Cns_1%40172.23.105.131%2Cns_1%40172.23.105.210%2Cns_1%40172.23.105.132
[2014-06-23 12:14:39,123] - [rest_client:1091] INFO - rebalance operation started
[2014-06-23 12:14:39,133] - [rest_client:1208] INFO - rebalance percentage : 0 %
[2014-06-23 12:14:44,148] - [rest_client:1208] INFO - rebalance percentage : 7.1484375 %
[2014-06-23 12:14:49,163] - [rest_client:1208] INFO - rebalance percentage : 12.0703125 %
[2014-06-23 12:14:54,174] - [rest_client:1208] INFO - rebalance percentage : 14.8828125 %
[2014-06-23 12:14:59,186] - [rest_client:1208] INFO - rebalance percentage : 22.265625 %
[2014-06-23 12:15:04,210] - [rest_client:1208] INFO - rebalance percentage : 28.59375 %
[2014-06-23 12:15:09,238] - [rest_client:1208] INFO - rebalance percentage : 30.8203125 %
[2014-06-23 12:15:14,251] - [rest_client:1208] INFO - rebalance percentage : 37.03125 %
[2014-06-23 12:15:19,265] - [rest_client:1208] INFO - rebalance percentage : 37.3828125 %
[2014-06-23 12:15:24,281] - [rest_client:1208] INFO - rebalance percentage : 44.296875 %
[2014-06-23 12:15:29,298] - [rest_client:1208] INFO - rebalance percentage : 50.0390625 %
[2014-06-23 12:15:34,314] - [rest_client:1208] INFO - rebalance percentage : 52.5 %
[2014-06-23 12:15:39,331] - [rest_client:1208] INFO - rebalance percentage : 59.765625 %
[2014-06-23 12:15:44,349] - [rest_client:1208] INFO - rebalance percentage : 67.3828125 %
[2014-06-23 12:15:49,366] - [rest_client:1208] INFO - rebalance percentage : 71.484375 %
[2014-06-23 12:15:54,384] - [rest_client:1208] INFO - rebalance percentage : 80.859375 %
[2014-06-23 12:15:59,400] - [rest_client:1208] INFO - rebalance percentage : 90.0390625 %
[2014-06-23 12:16:04,416] - [rest_client:1208] INFO - rebalance percentage : 96.7578125 %
[2014-06-23 12:16:09,432] - [rest_client:1208] INFO - rebalance percentage : 100 %
[2014-06-23 12:16:14,461] - [rest_client:1208] INFO - rebalance percentage : 100 %
[2014-06-23 12:16:19,478] - [rest_client:1164] INFO - rebalance progress took 100.354690075 seconds
[2014-06-23 12:16:19,481] - [rest_client:1165] INFO - sleep for 10 seconds after rebalance...