Description
as a continuation of the failed test in MB-7083
build 1941
new execution and test against fails
./testrunner -i /tmp/rebalance_regression.ini get-logs=True,wait_timeout=90 -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=3,num-swap=2,keys-count=1000000
http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/119/consoleFull
[ns_server:info,2012-11-06T6:06:32.319,ns_1@10.3.121.92:<0.3463.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.18320.33>,
{dict,7,16,16,8,80,48,
,
{{[],[],
[['ns_1@10.3.121.92'|
0.5793650793650793]],
[['ns_1@10.3.121.93'|0.59375]],
[['ns_1@10.3.121.94'|
0.6643835616438356]],
[['ns_1@10.3.121.95'|
0.5590386624869383]],
[['ns_1@10.3.121.96'|
0.4716417910447761]],
[['ns_1@10.3.121.97'|
0.4421157684630738]],
[['ns_1@10.3.121.98'|
0.3333333333333333]],
[],[],[],[],[],[],[]}}},
['ns_1@10.3.121.94',
'ns_1@10.3.121.92',
'ns_1@10.3.121.98',
'ns_1@10.3.121.96',
'ns_1@10.3.121.93',
'ns_1@10.3.121.97',
'ns_1@10.3.121.95'],
[],[]}
[ns_server:info,2012-11-06T6:06:32.668,ns_1@10.3.121.92:<0.19318.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/588">>
[ns_server:info,2012-11-06T6:06:32.974,ns_1@10.3.121.92:<0.19329.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/589">>
[ns_server:info,2012-11-06T6:06:33.156,ns_1@10.3.121.92:<0.19340.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/590">>
[ns_server:info,2012-11-06T6:06:33.513,ns_1@10.3.121.92:<0.19362.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/591">>
[ns_server:info,2012-11-06T6:06:33.668,ns_1@10.3.121.92:<0.19373.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/592">>
[ns_server:info,2012-11-06T6:06:33.813,ns_1@10.3.121.92:<0.19384.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/593">>
[ns_server:error,2012-11-06T6:06:34.009,ns_1@10.3.121.92:<0.11407.5>:ns_memcached:verify_report_long_call:297]call topkeys took too long: 680168 us
[ns_server:info,2012-11-06T6:06:34.028,ns_1@10.3.121.92:<0.19395.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/594">>
[ns_server:debug,2012-11-06T6:06:34.077,ns_1@10.3.121.92:<0.344.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.18067.36>,
{#Ref<0.0.200.188321>,exit,
{timeout,
{gen_server,call,
[<0.17521.36>,
,
30000]}},
[
,
,
]}}
[error_logger:error,2012-11-06T6:06:34.075,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server 'tap_replication_manager-bucket-1' terminating
-
- Last message in was
{change_vbucket_replication,567,undefined}
** When Server state == {state,"bucket-1",79650915,
[{'ns_1@10.3.121.93', [147,148,149,150,151,152,153,154,155,156,157, 158,159,160,161,162,163,164,165,166,167,168, 169,170,171,464,465,466,467,468,469,470,471, 756,757,758,759,760,761,762,763,764,765,766, 767,768,769,770,771,804,805,806,807,808,809, 810,811,816,817,818,819,862,863,864,865,866, 867,868,869]},
{'ns_1@10.3.121.94', [172,173,174,175,176,177,178,179,294,295,296, 297,298,299,300,301,302,303,304,305,306,307, 308,309,310,311,312,313,314,315,316,317]},
{'ns_1@10.3.121.95', [343,344,345,346,347,348,349,350,351,352,353, 354,355,356,357,358,359,360,361,362,363,364, 365,366,367,440,441,442,443,444,445,446,447, 448,449,661,662,663,664,665,666,692,693,694, 695,696,697,698,716,717,718,719,720,721,722, 723,724,725,726,727,728,729,730,731,812,813, 814,815,927,928,929,930,931,983,984,985,986, 987,988,989,990]},
{'ns_1@10.3.121.96', [222,223,224,225,226,227,228,229,567,568,569, 570,571,572,573,574,575,602,603,604,605,606, 607,608,609,820,821,822,823,824,825,826,827, 828,829,830,831,832,833,834,835,836,837,838, 839,840,841,842,843,844,886,887,888,889,890, 891,892,893,894]},
{'ns_1@10.3.121.97', [392,393,394,395,396,397,398,399,538,539,540, 541,542,543,544,545,732,733,734,735,736,737, 738,739,895,896,897,898,899,900,901,902,911, 912,913,914,915,916,917,918,919,920,921,922, 923,924,925,926,967,968,969,970,971,972,973, 974]},
{'ns_1@10.3.121.98', [270,271,272,273,274,275,276,277]}]}
** Reason for termination ==
** {timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570,571,572,573,574, 575,602,603,604,605,606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835,836,837,838,839,840, 841,842,843,844,886,887,888,889,890,891,892,893,894]},
30000]}}
[error_logger:error,2012-11-06T6:06:34.082,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: tap_replication_manager:init/1
pid: <0.11336.5>
registered_name: 'tap_replication_manager-bucket-1'
exception exit: {timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]},
30000]}}
in function gen_server:terminate/6
ancestors: ['single_bucket_sup-bucket-1',<0.11295.5>]
messages: []
links: [<0.11296.5>]
dictionary: []
trap_exit: false
status: running
heap_size: 317811
stack_size: 24
reductions: 14941478
neighbours:
[ns_server:info,2012-11-06T6:06:34.119,ns_1@10.3.121.92:<0.19406.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/595">>
[error_logger:error,2012-11-06T6:06:34.193,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,'single_bucket_sup-bucket-1'}
Context: child_terminated
Reason: {timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]},
30000]}}
Offender: [{pid,<0.11336.5>},
{name,{tap_replication_manager,"bucket-1"}},
{mfargs,{tap_replication_manager,start_link,["bucket-1"]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]
[error_logger:info,2012-11-06T6:06:34.195,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,'single_bucket_sup-bucket-1'}
started: [{pid,<0.19408.36>},
{name,{tap_replication_manager,"bucket-1"}},
{mfargs,
{tap_replication_manager,start_link,["bucket-1"]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]
[ns_server:info,2012-11-06T6:06:34.224,ns_1@10.3.121.92:janitor_agent-bucket-1<0.19413.36>:janitor_agent:read_flush_counter:764]Loading flushseq failed: {error,enoent}. Assuming it's equal to global config.
[ns_server:info,2012-11-06T6:06:34.225,ns_1@10.3.121.92:janitor_agent-bucket-1<0.19413.36>:janitor_agent:read_flush_counter_from_config:771]Initialized flushseq 0 from bucket config
[rebalance:debug,2012-11-06T6:06:34.226,ns_1@10.3.121.92:<0.10479.35>:janitor_agent:bulk_set_vbucket_state:398]bulk vbucket state change failed for:
[{'ns_1@10.3.121.92',
{'EXIT',
{{{timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570,571,572, 573,574,575,602,603,604,605,606,607,608,609,820,821, 822,823,824,825,826,827,828,829,830,831,832,833,834, 835,836,837,838,839,840,841,842,843,844,886,887,888, 889,890,891,892,893,894]},
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
{change_vbucket_replication,567,undefined},
{'janitor_agent-bucket-1','ns_1@10.3.121.92'},
infinity]}},
{gen_server,call,
[
{if_rebalance,<0.10479.35>,
{update_vbucket_state,567,replica,undefined,undefined}},
infinity]}}}}]
[error_logger:error,2012-11-06T6:06:34.226,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server 'janitor_agent-bucket-1' terminating
** Last message in was {if_rebalance,<0.10479.35>,
{update_vbucket_state,567,replica,undefined,
undefined}}
** When Server state == {state,"bucket-1",<0.10479.35>,#Ref<0.0.188.16601>,[], [active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,missing,missing,missing,missing, missing,missing,missing,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,replica,replica, replica,replica,replica,replica,replica, replica,missing,missing,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, active,active,active,active,active,active, active,active,active,active,active,active, active,active,replica,replica,replica,replica, replica,replica,replica,replica,missing, missing,missing,missing,missing,missing, missing,missing,active,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,active,active, active,active,active,active,active,active, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,active,active,active, active,active,active,active,active,active, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,missing, missing,missing,missing,missing,missing, missing,missing,active,active,active,active, active,active,active,active,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing], [undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined], 0}
** Reason for termination ==
** {{timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570,571,572,573,574, 575,602,603,604,605,606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835,836,837,838,839,840, 841,842,843,844,886,887,888,889,890,891,892,893,894]},
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
{change_vbucket_replication,567,undefined},
infinity]}}
[ns_server:info,2012-11-06T6:06:34.347,ns_1@10.3.121.92:<0.19421.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/596">>
[ns_server:info,2012-11-06T6:06:34.454,ns_1@10.3.121.92:ns_port_memcached<0.32588.3>:ns_port_server:log:171]memcached<0.32588.3>: Tue Nov 6 06:06:34.224138 PST 3: TAP (Producer) eq_tapq:replication_building_450_'ns_1@10.3.121.98' - disconnected, keep alive for 300 seconds
memcached<0.32588.3>: Tue Nov 6 06:06:34.229013 PST 3: TAP (Producer) eq_tapq:replication_building_450_'ns_1@10.3.121.95' - disconnected, keep alive for 300 seconds
[error_logger:error,2012-11-06T6:06:34.525,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: janitor_agent:init/1
pid: <0.11339.5>
registered_name: 'janitor_agent-bucket-1'
exception exit: {{timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569, 570,571,572,573,574,575,602,603,604,605, 606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835, 836,837,838,839,840,841,842,843,844,886, 887,888,889,890,891,892,893,894]},
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
{change_vbucket_replication,567,undefined},
infinity]}}
in function gen_server:terminate/6
ancestors: ['single_bucket_sup-bucket-1',<0.11295.5>]
messages: [{'$gen_call',
{<0.18042.36>,#Ref<0.0.200.199950>},
{if_rebalance,<0.10479.35>,
{get_replication_persistence_checkpoint_id,450}}}]
links: [<0.11296.5>]
dictionary: []
trap_exit: false
status: running
heap_size: 17711
stack_size: 24
reductions: 12271717
neighbours:
[error_logger:error,2012-11-06T6:06:34.568,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,'single_bucket_sup-bucket-1'}
Context: child_terminated
Reason: {{timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]},
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
{change_vbucket_replication,567,undefined},
infinity]}}
Offender: [{pid,<0.11339.5>},
{name,{janitor_agent,"bucket-1"}},
{mfargs,{janitor_agent,start_link,["bucket-1"]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]
[error_logger:info,2012-11-06T6:06:34.608,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,'single_bucket_sup-bucket-1'}
started: [{pid,<0.19413.36>},
{name,{janitor_agent,"bucket-1"}},
{mfargs,{janitor_agent,start_link,["bucket-1"]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]
[error_logger:error,2012-11-06T6:06:34.740,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,'ns_vbm_new_sup-bucket-1'}
Context: child_terminated
Reason: killed
Offender: [{pid,<0.17521.36>},
{name,{new_child_id,[222,223,224,225,226,227,228,229,567,
568,569,570,571,572,573,574,575,602,
603,604,605,606,607,608,609,820,821,
822,823,824,825,826,827,828,829,830,
831,832,833,834,835,836,837,838,839,
840,841,842,843,844,886,887,888,889,
890,891,892,893,894],
'ns_1@10.3.121.96'}},
{mfargs,{ebucketmigrator_srv,start_link,undefined}},
{restart_type,temporary},
{shutdown,60000},
{child_type,worker}]
[ns_server:info,2012-11-06T6:06:34.736,ns_1@10.3.121.92:<0.19442.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/597">>
[ns_server:info,2012-11-06T6:06:34.912,ns_1@10.3.121.92:<0.19453.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/598">>
[ns_server:info,2012-11-06T6:06:34.931,ns_1@10.3.121.92:<0.18074.36>:ns_replicas_builder_utils:kill_a_bunch_of_tap_names:59]Killed the following tap names on 'ns_1@10.3.121.92': [<<"replication_building_450_'ns_1@10.3.121.95'">>,
<<"replication_building_450_'ns_1@10.3.121.98'">>]
[error_logger:error,2012-11-06T6:06:34.981,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:mover/6
pid: <0.18042.36>
registered_name: []
exception exit: {{{timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569, 570,571,572,573,574,575,602,603,604,605, 606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835, 836,837,838,839,840,841,842,843,844,886, 887,888,889,890,891,892,893,894]},
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
{change_vbucket_replication,567,undefined},
infinity]}},
{gen_server,call,
[{'janitor_agent-bucket-1','ns_1@10.3.121.92'},
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]}
{if_rebalance,<0.10479.35>,
{get_replication_persistence_checkpoint_id,450}},
infinity]}}
in function gen_server:call/3
in call from janitor_agent:get_replication_persistence_checkpoint_id/4
in call from ns_single_vbucket_mover:mover_inner/6
in call from misc:try_with_maybe_ignorant_after/2
in call from ns_single_vbucket_mover:mover/6
ancestors: [<0.10479.35>,<0.18320.33>]
messages: [{'EXIT',<0.10479.35>,
{{bulk_set_vbucket_state_failed,
[{'ns_1@10.3.121.92',
{'EXIT',
{{{timeout,
{gen_server,call,
[<0.17521.36>,
,
{change_vbucket_replication,567,undefined}
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
,
{'janitor_agent-bucket-1','ns_1@10.3.121.92'}
infinity]}},
{gen_server,call,
[,
{janitor_agent,bulk_set_vbucket_state,4},
{if_rebalance,<0.10479.35>,
{update_vbucket_state,567,replica,undefined,
undefined}},
infinity]}}}}]},
[
{ns_vbucket_mover,update_replication_post_move,3},
{ns_vbucket_mover,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}}]
links: [<0.10479.35>,<0.19414.36>]
dictionary: [{cleanup_list,[<0.18074.36>]}]
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 3379
neighbours:
[user:info,2012-11-06T6:06:35.024,ns_1@10.3.121.92:<0.3463.0>:ns_orchestrator:handle_info:319]Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'ns_1@10.3.121.92',
{'EXIT',
{{{timeout,
{gen_server,call,
[<0.17521.36>,
{start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568, 569,570,571,572,573,574,575,602,603, 604,605,606,607,608,609,820,821,822, 823,824,825,826,827,828,829,830,831, 832,833,834,835,836,837,838,839,840, 841,842,843,844,886,887,888,889,890, 891,892,893,894]},
30000]}},
{gen_server,call,
['tap_replication_manager-bucket-1',
{change_vbucket_replication,567, undefined},
infinity]}},
{gen_server,call,
[{'janitor_agent-bucket-1', 'ns_1@10.3.121.92'},
{if_rebalance,<0.10479.35>,
{update_vbucket_state,567,replica,
undefined,undefined}},
infinity]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4},
{ns_vbucket_mover, update_replication_post_move,3}
,
{ns_vbucket_mover,handle_info,2}
,
{gen_server,handle_msg,5}
,
{proc_lib,init_p_do_apply,3}
]}
- Last message in was
{change_vbucket_replication,567,undefined}
Aleksey Kondratenko added a comment - 06/Nov/12 10:11 AM
It may be related and may indicate that original problem was not a deadlock in ebucketmigrator but some slowness in memcached. But anyway, please file new ticket.
Attachments
For Gerrit Dashboard: MB-7108 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
22383,2 | MB-7108: send downstream confirmation request async | master | ns_server | Status: MERGED | +2 | +1 |