Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
2.1.1
-
Security Level: Public
-
windows 2012 standard 64-bit
-
Windows 64-bit
-
Description
Environment:
7 windows server 2012 standard 64-bit (each with 4 core cpu, 9GB RAM and 150 GB storage in single regular hard drive)
1:172.23.105.184
2:172.23.105.185
3:172.23.105.186
4:172.23.105.187
5:172.23.105.188
6:172.23.105.189
7:172.23.105.190
2 windows server 2012 with the same spec above use to add in and swap node test
8:172.23.105.191
9:172.23.105.192
Cluster:
Create 7 nodes cluster installed couchbase server 2.1.1-766
Create 2 buckets with 1 replica: default and saslbucket (3.5GB per bucket)
Load 25 million items size from 128 bytes to 512 bytes to each bucket.
Continue to load until active resident reach 70% at each bucket.
Procedure to test:
Add node 191 to cluster ==> passed
Swap node 184 with node 192 ==> passed
Memcached crash on node 187 (will filed bug for this)
Auto failover node 190 and rebalance ==> failed
Tried rebalace few times later ==> falied
Error for rebalance failed:
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.30403.20>,
{badmatch,
[{'EXIT',
{timeout,
ns_orchestrator002 ns_1@172.23.105.185 21:55:06 - Wed Aug 14, 2013
<0.30396.20> exited with {unexpected_exit,
{'EXIT',<0.30403.20>,
{badmatch,
[{'EXIT',
{timeout,{gen_server,call, [<12838.5928.9>,had_backfill,30000]}
}}]}}} ns_vbucket_mover000 ns_1@172.23.105.185 21:54:16 - Wed Aug 14, 2013
Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'ns_1@172.23.105.185',
{'EXIT',
{{{{unexpected_reason,killed},
[
{gen_server,call,
['tap_replication_manager-saslbucket', {change_vbucket_replication,929, 'ns_1@172.23.105.186'},
infinity]}},
{gen_server,call,
[{'janitor_agent-saslbucket', 'ns_1@172.23.105.185'},
{if_rebalance,<0.2067.21>,
{update_vbucket_state,929,replica,
undefined,'ns_1@172.23.105.186'}},
infinity]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,on_move_done,2},{gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}
ns_orchestrator002 ns_1@172.23.105.185 23:52:08 - Wed Aug 14, 2013
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.6651.24>,
{badmatch,
[{'EXIT',
{timeout, {gen_server,call, [<12835.5816.18>,had_backfill, 30000]}}}]}}}
ns_orchestrator002 ns_1@172.23.105.185 00:41:11 - Thu Aug 15, 2013
<0.6642.24> exited with {unexpected_exit,
{'EXIT',<0.6651.24>,
{badmatch,
[{'EXIT',
{timeout, {gen_server,call, [<12835.5816.18>,had_backfill,30000]}}}]}}} ns_vbucket_mover000 ns_1@172.23.105.185 00:41:11 - Thu Aug 15, 2013
Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'ns_1@172.23.105.192',
{'EXIT',
{{{{unexpected_reason,killed},
[{misc,executing_on_new_process,1}
,
{tap_replication_manager, change_vbucket_filter,4},
{tap_replication_manager, '-do_set_incoming_replication_map/3-lc$^2/1-2-', 2},
{tap_replication_manager, do_set_incoming_replication_map,3},
{tap_replication_manager,handle_call,3},
{gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]},{gen_server,call,
['tap_replication_manager-saslbucket', {change_vbucket_replication,853, 'ns_1@172.23.105.188'},
infinity]}},
{gen_server,call,
[{'janitor_agent-saslbucket', 'ns_1@172.23.105.192'},
{if_rebalance,<0.16519.24>,
{update_vbucket_state,853,replica,
undefined,'ns_1@172.23.105.188'}},
infinity]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,on_move_done,2},{gen_server,handle_msg,5}
,
{proc_lib,init_p_do_apply,3}]}
ns_orchestrator002 ns_1@172.23.105.185 00:58:42 - Thu Aug 15, 2013
Attachments
Issue Links
- relates to
-
MB-8865 rebalance failed with error "badmatch,[{<17897.4408.1>,noproc}...
- Closed