Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8094

[Doc'd in 2.0.2]1.8.1 & 2.0.2 mixed cluster:Rebalance exited with reason {badarg, [{ns_rebalancer, '-wait_for_memcached/3-lc$^0/1-0-',2}, {ns_rebalancer,wait_for_memcached,

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None

      Description

      http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-new-rebalance-mixed-cluster/62/consoleFull
      ./testrunner -i /tmp/rebalance_in.ini get-logs=True,wait_timeout=180,GROUP=P0,EXCLUDE_GROUP=FROM_2_0,get-cbcollect-infrebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... o=True -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,GROUP=IN;P0

      mixed suite/cluster:
      1.8.1-937-rel
      10.3.3.92
      10.3.3.94
      10.3.3.93

      2.0.2-764-rel
      10.3.3.99
      10.3.3.91
      10.3.3.82
      10.3.3.97

      add to 10.3.3.92(1.8.1) 1*1.8.1(10.3.3.93) and 2*2.0.2(10.3.3.82, 10.3.3.99)

      2013-04-14 09:06:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.3.92%2Cns_1%4010.3.3.82%2Cns_1%4010.3.3.99%2Cns_1%4010.3.3.93
      2013-04-14 09:06:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
      2013-04-14 09:06:38 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0 %
      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      2013-04-14 09:06:48 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] Latest logs from UI:
      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.99', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. Will try to take over.", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956005954.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.82', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. Will try to take over.", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956005954.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.93', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.93\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1365956004357.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] {u'node': u'ns_1@10.3.3.92', u'code': 2, u'text': u"Rebalance exited with reason {badarg,\n [

      {ns_rebalancer,\n '-wait_for_memcached/3-lc$^0/1-0-',2}

      ,\n

      {ns_rebalancer,wait_for_memcached,3}

      ,\n

      {ns_rebalancer,'-rebalance/3-fun-0-',5}

      ,\n

      {lists,foreach,2}

      ,\n

      {ns_rebalancer,rebalance,3}

      ]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1365956004324.0, u'type': u'info'}
      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.99', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. But I won't try to take over since rebalance seems to be running", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956003954.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.82', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. But I won't try to take over since rebalance seems to be running", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956003954.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.82', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.82\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1365956003367.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.99', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.99\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1365956003359.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.92', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1365956003298.0, u'type': u'info'}

      2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'node': u'ns_1@10.3.3.92', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.3.92','ns_1@10.3.3.82',\n 'ns_1@10.3.3.99','ns_1@10.3.3.93'], EjectNodes = []\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1365956003257.0, u'type': u'info'}
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

          • Assignee:
            andreibaranouski Andrei Baranouski
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes