Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
3.1.0
-
Security Level: Public
-
None
-
centos 6.5 64-bit, 8 core cpu, 30 GB RAM, 80 GB SSD, 10 Git network
-
Untriaged
-
Centos 64-bit
-
-
Unknown
-
KV: Sep 14 - Oct 2
Description
Install cb server 3.1.0-1776 on 7 nodes
Create cluster of 7 nodes
Create defautl bucket with 2 replica
Load 119 million items with size 100 bytes to default bucket (no view and 100% set) Front load is 250K/second
Install cb server 3.1.0-1776 on other 5 nodes
Add new 5 nodes to cluster and remove 5 nodes (swap rebalance)
Rebalance failed with error memcached disconnect
Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
{error,
timeout}} ns_memcached000 ns_1@10.156.27.78 20:33:33 - Tue Aug 18, 2015
Bucket "default" loaded on node 'ns_1@10.156.27.78' in 0 seconds. ns_memcached000 ns_1@10.156.27.78 20:26:47 - Tue Aug 18, 2015
Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
{error,
timeout}} ns_memcached000 ns_1@10.156.27.78 20:26:46 - Tue Aug 18, 2015
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.29088.218>,
{dcp_wait_for_data_move_failed,"default",137,
'ns_1@10.11.196.105',
['ns_1@10.45.75.76','ns_1@10.156.27.78'],
{error,no_stats_for_this_vbucket}}}}
ns_orchestrator002 ns_1@10.156.27.78 20:23:46 - Tue Aug 18, 2015
<0.28331.218> exited with {unexpected_exit,
{'EXIT',<0.29088.218>,
{dcp_wait_for_data_move_failed,"default",
137,'ns_1@10.11.196.105',
['ns_1@10.45.75.76','ns_1@10.156.27.78'],
{error,no_stats_for_this_vbucket}}}} ns_vbucket_mover000 ns_1@10.156.27.78 20:23:46 - Tue Aug 18, 2015
Bucket "default" loaded on node 'ns_1@10.156.27.78' in 0 seconds. ns_memcached000 ns_1@10.156.27.78 20:19:04 - Tue Aug 18, 2015
Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
{error,
timeout}} ns_memcached000 ns_1@10.156.27.78 20:19:03 - Tue Aug 18, 2015
Bucket "default" loaded on node 'ns_1@10.156.27.78' in 0 seconds. ns_memcached000 ns_1@10.156.27.78 20:09:02 - Tue Aug 18, 2015
Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
{error,
timeout}} ns_memcached000 ns_1@10.156.27.78 20:09:02 - Tue Aug 18, 2015
Bucket "default" loaded on node 'ns_1@10.156.46.169' in 0 seconds. ns_memcached000 ns_1@10.156.46.169 19:16:11 - Tue Aug 18, 2015
Deleting old data files of bucket "default" ns_storage_conf000 ns_1@10.156.46.169 19:16:10 - Tue Aug 18, 2015
Bucket "default" loaded on node 'ns_1@10.229.83.110' in 0 seconds. ns_memcached000 ns_1@10.229.83.110 19:16:09 - Tue Aug 18, 2015
Deleting old data files of bucket "default" ns_storage_conf000 ns_1@10.229.83.110 19:16:08 - Tue Aug 18, 2015
Bucket "default" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@10.156.27.78 19:16:02 - Tue Aug 18, 2015
Bucket "default" loaded on node 'ns_1@10.45.75.76' in 0 seconds. ns_memcached000 ns_1@10.45.75.76 19:16:01 - Tue Aug 18, 2015
Deleting old data files of bucket "default" ns_storage_conf000 ns_1@10.45.75.76 19:16:01 - Tue Aug 18, 2015
Started rebalancing bucket default ns_rebalancer000 ns_1@10.156.27.78 19:15:58 - Tue Aug 18, 2015
Starting rebalance, KeepNodes = ['ns_1@10.11.196.105','ns_1@10.156.27.78',
'ns_1@10.45.75.76','ns_1@10.141.207.54',
'ns_1@10.154.33.77','ns_1@10.156.46.169',
'ns_1@10.229.83.110'], EjectNodes = ['ns_1@10.157.170.23',
'ns_1@10.152.47.238',
'ns_1@10.182.90.53',
'ns_1@10.141.15.231',
'ns_1@10.182.89.250'], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@10.156.27.78 19:15:57 - Tue Aug 18, 2015
I will try rebalance again and update this ticket