Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8109

Rebalance exited with reason {unexpected_exit,.."Partition 375 not in active nor passive set"

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • Major
    • 2.1.0
    • 2.1.0
    • view-engine
    • Security Level: Public
    • None
    • CentOS release 5.8, 2.0.2-764-re

    Description

      http://qa.hq.northscale.net/view/2.0.1/job/ubuntu-64-2.0-new-rebalance-tests-P0/415/console
      ./testrunner -i /tmp/rebalance-tests.ini wait_timeout=180,GROUP=P0,get-cbcollect-info=True,stop-on-failure=True -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ddoc_compaction,items=500000,max_verify=100000,fragmentation_value=60,GROUP=OUT;BASIC;COMPACTION;P0;FROM_2_0

      test with ddoc compaction:
      default bucket with 500000 items, 1ddoc*5views

      [2013-04-16 05:50:02,412] - [task:1753] INFO - design doc ddoc1 is compacting
      [2013-04-16 05:50:07,490] - [task:1748] INFO - stats compaction: (1,45.2233728939)
      [2013-04-16 05:50:07,491] - [task:1756] INFO - compactor was run, compaction revision was changed on 1
      [2013-04-16 05:50:07,491] - [task:1759] INFO - fragmentation went from 69 to 45
      [2013-04-16 05:50:09,511] - [rest_client:940] INFO - rebalance percentage : 15.9197945499 %
      [2013-04-16 05:50:19,525] - [rest_client:940] INFO - rebalance percentage : 33.5959873768 %
      [2013-04-16 05:50:29,534] - [rest_client:940] INFO - rebalance percentage : 52.8221587537 %
      [2013-04-16 05:50:39,542] - [rest_client:940] INFO - rebalance percentage : 66.4929596916 %
      [2013-04-16 05:50:50,558] - [rest_client:940] INFO - rebalance percentage : 75.6601053108 %
      [2013-04-16 05:51:01,563] - [rest_client:923] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      [2013-04-16 05:51:01,563] - [rest_client:924] INFO - Latest logs from UI:
      [2013-04-16 05:51:01,659] - [rest_client:925] ERROR - {u'node': u'ns_1@10.5.2.13', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n {\'EXIT\',<0.9415.34>,\n {{{{badmatch,\n {error,\n

      {error,\n <<"Partition 375 not in active nor passive set">>}

      }},\n [

      {capi_set_view_manager,handle_call,3},\n {gen_server,handle_msg,5},\n {gen_server,init_it,6},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [\'capi_set_view_manager-default\',\n {wait_index_updated,375},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\'ns_1@10.5.2.13\'},\n {if_rebalance,<0.31446.33>,\n {wait_index_updated,375}},\n infinity]}}}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1366117462189, u'type': u'info'}
      [2013-04-16 05:51:01,659] - [rest_client:925] ERROR - {u'node': u'ns_1@10.5.2.13', u'code': 0, u'text': u'<0.9278.34> exited with {unexpected_exit,\n {\'EXIT\',<0.9415.34>,\n {{badmatch,\n {error,\n {error,\n <<"Partition 375 not in active nor passive set">>},\n [{capi_set_view_manager,handle_call,3}

      ,\n

      {gen_server,handle_msg,5}

      ,\n

      {gen_server,init_it,6}

      ,\n

      {proc_lib,init_p_do_apply,3}

      ]},\n {gen_server,call,\n [\'capi_set_view_manager-default\',\n

      {wait_index_updated,375}

      ,\n infinity]}},\n {gen_server,call,\n [

      {\'janitor_agent-default\',\'ns_1@10.5.2.13\'}

      ,\n {if_rebalance,<0.31446.33>,\n {wait_index_updated,375}},\n infinity]}}}}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1366117462002, u'type': u'critical'}
      [2013-04-16 05:51:01,660] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.5.2.14', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` completed.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117432811, u'type': u'info'}

      [2013-04-16 05:51:01,660] - [rest_client:925] ERROR - {u'node': u'ns_1@10.3.121.66', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` failed:

      {index_compactor_died_too_soon,\n <<"default">>,\n <<"_design/ddoc1">>,\n main}

      . See logs for detailed reason.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117427129, u'type': u'critical'}
      [2013-04-16 05:51:01,660] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.3.121.64', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` completed.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117425016, u'type': u'info'}

      [2013-04-16 05:51:01,661] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.5.2.15', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` completed.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117421694, u'type': u'info'}

      [2013-04-16 05:51:01,661] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.3.121.63', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` completed.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117408988, u'type': u'info'}

      [2013-04-16 05:51:01,661] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.5.2.13', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1366117402503, u'type': u'info'}

      [2013-04-16 05:51:01,661] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.5.2.13', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1366117402122, u'type': u'info'}

      [2013-04-16 05:51:01,661] - [rest_client:925] ERROR -

      {u'node': u'ns_1@10.5.2.13', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.5.2.14','ns_1@10.5.2.15',\n 'ns_1@10.3.121.63','ns_1@10.3.121.69',\n 'ns_1@10.5.2.13','ns_1@10.3.121.64'], EjectNodes = ['ns_1@10.3.121.66']\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1366117402023, u'type': u'info'}

      Attachments

        For Gerrit Dashboard: MB-8109
        # Subject Branch Project Status CR V

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty