Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7108

Rebalance exited with reason {{bulk_set_vbucket_state_failed after crash in tap_replication_manager:init/1 registered_name: 'tap_replication_manager-bucket-1' exception exit: {timeout, {gen_server ,{start_vbucket_filter_change

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.0
    • 2.0-beta-2
    • ns_server
    • Security Level: Public
    • None

    Description

      as a continuation of the failed test in MB-7083

      build 1941
      new execution and test against fails
      ./testrunner -i /tmp/rebalance_regression.ini get-logs=True,wait_timeout=90 -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=3,num-swap=2,keys-count=1000000
      http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/119/consoleFull

      [ns_server:info,2012-11-06T6:06:32.319,ns_1@10.3.121.92:<0.3463.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.18320.33>,
      {dict,7,16,16,8,80,48,

      {[],[],[],[],[],[],[],[],[],[],[],[], [],[],[],[]}

      ,
      {{[],[],
      [['ns_1@10.3.121.92'|
      0.5793650793650793]],
      [['ns_1@10.3.121.93'|0.59375]],
      [['ns_1@10.3.121.94'|
      0.6643835616438356]],
      [['ns_1@10.3.121.95'|
      0.5590386624869383]],
      [['ns_1@10.3.121.96'|
      0.4716417910447761]],
      [['ns_1@10.3.121.97'|
      0.4421157684630738]],
      [['ns_1@10.3.121.98'|
      0.3333333333333333]],
      [],[],[],[],[],[],[]}}},
      ['ns_1@10.3.121.94',
      'ns_1@10.3.121.92',
      'ns_1@10.3.121.98',
      'ns_1@10.3.121.96',
      'ns_1@10.3.121.93',
      'ns_1@10.3.121.97',
      'ns_1@10.3.121.95'],
      [],[]}
      [ns_server:info,2012-11-06T6:06:32.668,ns_1@10.3.121.92:<0.19318.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/588">>
      [ns_server:info,2012-11-06T6:06:32.974,ns_1@10.3.121.92:<0.19329.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/589">>
      [ns_server:info,2012-11-06T6:06:33.156,ns_1@10.3.121.92:<0.19340.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/590">>
      [ns_server:info,2012-11-06T6:06:33.513,ns_1@10.3.121.92:<0.19362.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/591">>
      [ns_server:info,2012-11-06T6:06:33.668,ns_1@10.3.121.92:<0.19373.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/592">>
      [ns_server:info,2012-11-06T6:06:33.813,ns_1@10.3.121.92:<0.19384.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/593">>
      [ns_server:error,2012-11-06T6:06:34.009,ns_1@10.3.121.92:<0.11407.5>:ns_memcached:verify_report_long_call:297]call topkeys took too long: 680168 us
      [ns_server:info,2012-11-06T6:06:34.028,ns_1@10.3.121.92:<0.19395.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/594">>
      [ns_server:debug,2012-11-06T6:06:34.077,ns_1@10.3.121.92:<0.344.0>:ns_process_registry:handle_info:98]Got exit msg: {'EXIT',<0.18067.36>,
      {#Ref<0.0.200.188321>,exit,
      {timeout,
      {gen_server,call,
      [<0.17521.36>,

      {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]}

      ,
      30000]}},
      [

      {gen_server,call,3}

      ,

      {ns_vbm_new_sup, '-perform_vbucket_filter_change/6-fun-1-',7}

      ,

      {misc,'-executing_on_new_process/1-fun-0-',3}

      ]}}
      [error_logger:error,2012-11-06T6:06:34.075,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server 'tap_replication_manager-bucket-1' terminating

        • Last message in was {change_vbucket_replication,567,undefined}
          ** When Server state == {state,"bucket-1",79650915,
          [{'ns_1@10.3.121.93', [147,148,149,150,151,152,153,154,155,156,157, 158,159,160,161,162,163,164,165,166,167,168, 169,170,171,464,465,466,467,468,469,470,471, 756,757,758,759,760,761,762,763,764,765,766, 767,768,769,770,771,804,805,806,807,808,809, 810,811,816,817,818,819,862,863,864,865,866, 867,868,869]},
          {'ns_1@10.3.121.94', [172,173,174,175,176,177,178,179,294,295,296, 297,298,299,300,301,302,303,304,305,306,307, 308,309,310,311,312,313,314,315,316,317]},
          {'ns_1@10.3.121.95', [343,344,345,346,347,348,349,350,351,352,353, 354,355,356,357,358,359,360,361,362,363,364, 365,366,367,440,441,442,443,444,445,446,447, 448,449,661,662,663,664,665,666,692,693,694, 695,696,697,698,716,717,718,719,720,721,722, 723,724,725,726,727,728,729,730,731,812,813, 814,815,927,928,929,930,931,983,984,985,986, 987,988,989,990]},
          {'ns_1@10.3.121.96', [222,223,224,225,226,227,228,229,567,568,569, 570,571,572,573,574,575,602,603,604,605,606, 607,608,609,820,821,822,823,824,825,826,827, 828,829,830,831,832,833,834,835,836,837,838, 839,840,841,842,843,844,886,887,888,889,890, 891,892,893,894]},
          {'ns_1@10.3.121.97', [392,393,394,395,396,397,398,399,538,539,540, 541,542,543,544,545,732,733,734,735,736,737, 738,739,895,896,897,898,899,900,901,902,911, 912,913,914,915,916,917,918,919,920,921,922, 923,924,925,926,967,968,969,970,971,972,973, 974]},
          {'ns_1@10.3.121.98', [270,271,272,273,274,275,276,277]}]}
          ** Reason for termination ==
          ** {timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570,571,572,573,574, 575,602,603,604,605,606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835,836,837,838,839,840, 841,842,843,844,886,887,888,889,890,891,892,893,894]},
          30000]}}

          [error_logger:error,2012-11-06T6:06:34.082,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================CRASH REPORT=========================
          crasher:
          initial call: tap_replication_manager:init/1
          pid: <0.11336.5>
          registered_name: 'tap_replication_manager-bucket-1'
          exception exit: {timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]},
          30000]}}
          in function gen_server:terminate/6
          ancestors: ['single_bucket_sup-bucket-1',<0.11295.5>]
          messages: []
          links: [<0.11296.5>]
          dictionary: []
          trap_exit: false
          status: running
          heap_size: 317811
          stack_size: 24
          reductions: 14941478
          neighbours:

          [ns_server:info,2012-11-06T6:06:34.119,ns_1@10.3.121.92:<0.19406.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/595">>
          [error_logger:error,2012-11-06T6:06:34.193,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================SUPERVISOR REPORT=========================
          Supervisor: {local,'single_bucket_sup-bucket-1'}
          Context: child_terminated
          Reason: {timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]},
          30000]}}
          Offender: [{pid,<0.11336.5>},
          {name,{tap_replication_manager,"bucket-1"}},
          {mfargs,{tap_replication_manager,start_link,["bucket-1"]}},
          {restart_type,permanent},
          {shutdown,1000},
          {child_type,worker}]


          [error_logger:info,2012-11-06T6:06:34.195,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================PROGRESS REPORT=========================
          supervisor: {local,'single_bucket_sup-bucket-1'}
          started: [{pid,<0.19408.36>},
          {name,{tap_replication_manager,"bucket-1"}},
          {mfargs,
          {tap_replication_manager,start_link,["bucket-1"]}},
          {restart_type,permanent},
          {shutdown,1000},
          {child_type,worker}]

          [ns_server:info,2012-11-06T6:06:34.224,ns_1@10.3.121.92:janitor_agent-bucket-1<0.19413.36>:janitor_agent:read_flush_counter:764]Loading flushseq failed: {error,enoent}. Assuming it's equal to global config.
          [ns_server:info,2012-11-06T6:06:34.225,ns_1@10.3.121.92:janitor_agent-bucket-1<0.19413.36>:janitor_agent:read_flush_counter_from_config:771]Initialized flushseq 0 from bucket config
          [rebalance:debug,2012-11-06T6:06:34.226,ns_1@10.3.121.92:<0.10479.35>:janitor_agent:bulk_set_vbucket_state:398]bulk vbucket state change failed for:
          [{'ns_1@10.3.121.92',
          {'EXIT',
          {{{timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570,571,572, 573,574,575,602,603,604,605,606,607,608,609,820,821, 822,823,824,825,826,827,828,829,830,831,832,833,834, 835,836,837,838,839,840,841,842,843,844,886,887,888, 889,890,891,892,893,894]},
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',
          {change_vbucket_replication,567,undefined}

          ,
          infinity]}},
          {gen_server,call,
          [

          {'janitor_agent-bucket-1','ns_1@10.3.121.92'},
          {if_rebalance,<0.10479.35>,
          {update_vbucket_state,567,replica,undefined,undefined}},
          infinity]}}}}]
          [error_logger:error,2012-11-06T6:06:34.226,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server 'janitor_agent-bucket-1' terminating
          ** Last message in was {if_rebalance,<0.10479.35>,
          {update_vbucket_state,567,replica,undefined,
          undefined}}
          ** When Server state == {state,"bucket-1",<0.10479.35>,#Ref<0.0.188.16601>,[], [active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,missing,missing,missing,missing, missing,missing,missing,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,replica,replica, replica,replica,replica,replica,replica, replica,missing,missing,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, active,active,active,active,active,active, active,active,active,active,active,active, active,active,replica,replica,replica,replica, replica,replica,replica,replica,missing, missing,missing,missing,missing,missing, missing,missing,active,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,active,active, active,active,active,active,active,active, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,active,active,active, active,active,active,active,active,active, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,replica,replica,replica,replica, replica,replica,replica,replica,missing, missing,missing,missing,missing,missing, missing,missing,active,active,active,active, active,active,active,active,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing], [undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined], 0}
          ** Reason for termination ==
          ** {{timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570,571,572,573,574, 575,602,603,604,605,606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835,836,837,838,839,840, 841,842,843,844,886,887,888,889,890,891,892,893,894]},
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',
          {change_vbucket_replication,567,undefined},
          infinity]}}

          [ns_server:info,2012-11-06T6:06:34.347,ns_1@10.3.121.92:<0.19421.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/596">>
          [ns_server:info,2012-11-06T6:06:34.454,ns_1@10.3.121.92:ns_port_memcached<0.32588.3>:ns_port_server:log:171]memcached<0.32588.3>: Tue Nov 6 06:06:34.224138 PST 3: TAP (Producer) eq_tapq:replication_building_450_'ns_1@10.3.121.98' - disconnected, keep alive for 300 seconds
          memcached<0.32588.3>: Tue Nov 6 06:06:34.229013 PST 3: TAP (Producer) eq_tapq:replication_building_450_'ns_1@10.3.121.95' - disconnected, keep alive for 300 seconds

          [error_logger:error,2012-11-06T6:06:34.525,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================CRASH REPORT=========================
          crasher:
          initial call: janitor_agent:init/1
          pid: <0.11339.5>
          registered_name: 'janitor_agent-bucket-1'
          exception exit: {{timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569, 570,571,572,573,574,575,602,603,604,605, 606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835, 836,837,838,839,840,841,842,843,844,886, 887,888,889,890,891,892,893,894]},
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',
          {change_vbucket_replication,567,undefined},
          infinity]}}
          in function gen_server:terminate/6
          ancestors: ['single_bucket_sup-bucket-1',<0.11295.5>]
          messages: [{'$gen_call',
          {<0.18042.36>,#Ref<0.0.200.199950>},
          {if_rebalance,<0.10479.35>,
          {get_replication_persistence_checkpoint_id,450}}}]
          links: [<0.11296.5>]
          dictionary: []
          trap_exit: false
          status: running
          heap_size: 17711
          stack_size: 24
          reductions: 12271717
          neighbours:

          [error_logger:error,2012-11-06T6:06:34.568,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================SUPERVISOR REPORT=========================
          Supervisor: {local,'single_bucket_sup-bucket-1'}
          Context: child_terminated
          Reason: {{timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]},
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',
          {change_vbucket_replication,567,undefined},
          infinity]}}
          Offender: [{pid,<0.11339.5>},
          {name,{janitor_agent,"bucket-1"}},
          {mfargs,{janitor_agent,start_link,["bucket-1"]}},
          {restart_type,permanent},
          {shutdown,brutal_kill},
          {child_type,worker}]


          [error_logger:info,2012-11-06T6:06:34.608,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================PROGRESS REPORT=========================
          supervisor: {local,'single_bucket_sup-bucket-1'}
          started: [{pid,<0.19413.36>},
          {name,{janitor_agent,"bucket-1"}},
          {mfargs,{janitor_agent,start_link,["bucket-1"]}},
          {restart_type,permanent},
          {shutdown,brutal_kill},
          {child_type,worker}]

          [error_logger:error,2012-11-06T6:06:34.740,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================SUPERVISOR REPORT=========================
          Supervisor: {local,'ns_vbm_new_sup-bucket-1'}
          Context: child_terminated
          Reason: killed
          Offender: [{pid,<0.17521.36>},
          {name,{new_child_id,[222,223,224,225,226,227,228,229,567,
          568,569,570,571,572,573,574,575,602,
          603,604,605,606,607,608,609,820,821,
          822,823,824,825,826,827,828,829,830,
          831,832,833,834,835,836,837,838,839,
          840,841,842,843,844,886,887,888,889,
          890,891,892,893,894],
          'ns_1@10.3.121.96'}},
          {mfargs,{ebucketmigrator_srv,start_link,undefined}},
          {restart_type,temporary},
          {shutdown,60000},
          {child_type,worker}]


          [ns_server:info,2012-11-06T6:06:34.736,ns_1@10.3.121.92:<0.19442.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/597">>
          [ns_server:info,2012-11-06T6:06:34.912,ns_1@10.3.121.92:<0.19453.36>:compaction_daemon:spawn_vbucket_compactor:639]Compacting <<"bucket-1/598">>
          [ns_server:info,2012-11-06T6:06:34.931,ns_1@10.3.121.92:<0.18074.36>:ns_replicas_builder_utils:kill_a_bunch_of_tap_names:59]Killed the following tap names on 'ns_1@10.3.121.92': [<<"replication_building_450_'ns_1@10.3.121.95'">>,
          <<"replication_building_450_'ns_1@10.3.121.98'">>]
          [error_logger:error,2012-11-06T6:06:34.981,ns_1@10.3.121.92:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
          =========================CRASH REPORT=========================
          crasher:
          initial call: ns_single_vbucket_mover:mover/6
          pid: <0.18042.36>
          registered_name: []
          exception exit: {{{timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569, 570,571,572,573,574,575,602,603,604,605, 606,607,608,609,820,821,822,823,824,825, 826,827,828,829,830,831,832,833,834,835, 836,837,838,839,840,841,842,843,844,886, 887,888,889,890,891,892,893,894]},
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',
          {change_vbucket_replication,567,undefined},
          infinity]}},
          {gen_server,call,
          [{'janitor_agent-bucket-1','ns_1@10.3.121.92'}

          ,
          {if_rebalance,<0.10479.35>,
          {get_replication_persistence_checkpoint_id,450}},
          infinity]}}
          in function gen_server:call/3
          in call from janitor_agent:get_replication_persistence_checkpoint_id/4
          in call from ns_single_vbucket_mover:mover_inner/6
          in call from misc:try_with_maybe_ignorant_after/2
          in call from ns_single_vbucket_mover:mover/6
          ancestors: [<0.10479.35>,<0.18320.33>]
          messages: [{'EXIT',<0.10479.35>,
          {{bulk_set_vbucket_state_failed,
          [{'ns_1@10.3.121.92',
          {'EXIT',
          {{{timeout,
          {gen_server,call,
          [<0.17521.36>,

          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568,569,570, 571,572,573,574,575,602,603,604,605,606,607, 608,609,820,821,822,823,824,825,826,827,828, 829,830,831,832,833,834,835,836,837,838,839, 840,841,842,843,844,886,887,888,889,890,891, 892,893,894]}

          ,
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',

          {change_vbucket_replication,567,undefined}

          ,
          infinity]}},
          {gen_server,call,
          [

          {'janitor_agent-bucket-1','ns_1@10.3.121.92'}

          ,
          {if_rebalance,<0.10479.35>,
          {update_vbucket_state,567,replica,undefined,
          undefined}},
          infinity]}}}}]},
          [

          {janitor_agent,bulk_set_vbucket_state,4},
          {ns_vbucket_mover,update_replication_post_move,3},
          {ns_vbucket_mover,handle_info,2},
          {gen_server,handle_msg,5},
          {proc_lib,init_p_do_apply,3}]}}]
          links: [<0.10479.35>,<0.19414.36>]
          dictionary: [{cleanup_list,[<0.18074.36>]}]
          trap_exit: true
          status: running
          heap_size: 2584
          stack_size: 24
          reductions: 3379
          neighbours:

          [user:info,2012-11-06T6:06:35.024,ns_1@10.3.121.92:<0.3463.0>:ns_orchestrator:handle_info:319]Rebalance exited with reason {{bulk_set_vbucket_state_failed,
          [{'ns_1@10.3.121.92',
          {'EXIT',
          {{{timeout,
          {gen_server,call,
          [<0.17521.36>,
          {start_vbucket_filter_change, [222,223,224,225,226,227,228,229,568, 569,570,571,572,573,574,575,602,603, 604,605,606,607,608,609,820,821,822, 823,824,825,826,827,828,829,830,831, 832,833,834,835,836,837,838,839,840, 841,842,843,844,886,887,888,889,890, 891,892,893,894]},
          30000]}},
          {gen_server,call,
          ['tap_replication_manager-bucket-1',
          {change_vbucket_replication,567, undefined},
          infinity]}},
          {gen_server,call,
          [{'janitor_agent-bucket-1', 'ns_1@10.3.121.92'},
          {if_rebalance,<0.10479.35>,
          {update_vbucket_state,567,replica,
          undefined,undefined}},
          infinity]}}}}]},
          [{janitor_agent,bulk_set_vbucket_state,4}

          ,

          {ns_vbucket_mover, update_replication_post_move,3}

          ,

          {ns_vbucket_mover,handle_info,2}

          ,

          {gen_server,handle_msg,5}

          ,

          {proc_lib,init_p_do_apply,3}

          ]}

      Aleksey Kondratenko added a comment - 06/Nov/12 10:11 AM
      It may be related and may indicate that original problem was not a deadlock in ebucketmigrator but some slowness in memcached. But anyway, please file new ticket.

      https://s3.amazonaws.com/bugdb/jira/MB-7083/G34ku23lk/28becc6d-a107-4d1d-bee6-983cd14b5f96-10.3.121.92-diag.txt.gz

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty