Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11648

[Ubuntu] Online upgrade failed 2.5.0-1059 -> 3.0.0-918 due to rebalance failed {badmatch,wrong_rebalancer_pid}

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 3.0
    • 3.0
    • ns_server
    • Security Level: Public
    • None
    • Online upgrade 2.5.0-1059 -> 3.0.0-918

      Ubuntu 12.04

    Description

      [Test]
      http://qa.hq.northscale.net/job/ubuntu_x64--36_01--XDCR_upgrade-P1/18/consoleFull

      [Test Error]
      [2014-07-03 06:03:43,427] - [rest_client:1399] INFO - Node versions in cluster [u'2.5.0-1059-rel-enterprise', u'2.5.0-1059-rel-enterprise']
      [2014-07-03 06:03:43,442] - [rest_client:1399] INFO - Node versions in cluster [u'3.0.0-918-rel-enterprise']
      [2014-07-03 06:03:44,098] - [task:284] INFO - adding node 10.3.3.199:8091 to cluster
      [2014-07-03 06:03:44,098] - [rest_client:933] INFO - adding remote node @10.3.3.199:8091 to this cluster @10.3.3.240:8091
      [2014-07-03 06:03:46,289] - [rest_client:1087] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.3.199%2Cns_1%4010.3.3.240%2Cns_1%4010.3.3.218
      [2014-07-03 06:03:55,698] - [rest_client:1091] INFO - rebalance operation started
      [2014-07-03 06:03:55,758] - [rest_client:1208] INFO - rebalance percentage : 0 %
      [2014-07-03 06:04:05,775] - [rest_client:1192] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      [2014-07-03 06:04:05,845] - [rest_client:1999] INFO - Latest logs from UI on 10.3.3.240:
      [2014-07-03 06:04:05,846] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.3.3.199', u'code': 2, u'text': u'Rebalance exited with reason

      {badmatch,wrong_rebalancer_pid}\n', u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:55.085Z', u'module': u'ns_orchestrator', u'tstamp': 1404392635085, u'type': u'info'}
      [2014-07-03 06:04:05,847] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u'<0.1307.0> exited with {badmatch,wrong_rebalancer_pid}

      ', u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:55.084Z', u'module': u'ns_vbucket_mover', u'tstamp': 1404392635084, u'type': u'critical'}
      [2014-07-03 06:04:05,848] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:53.474Z', u'module': u'ns_vbucket_mover', u'tstamp': 1404392633474, u'type': u'info'}

      [2014-07-03 06:04:05,849] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.199\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:52.766Z', u'module': u'ns_memcached', u'tstamp': 1404392632766, u'type': u'info'}

      [2014-07-03 06:04:05,849] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:51.947Z', u'module': u'ns_rebalancer', u'tstamp': 1404392631947, u'type': u'info'}

      [2014-07-03 06:04:05,849] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.3.199','ns_1@10.3.3.240',\n 'ns_1@10.3.3.218'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:51.642Z', u'module': u'ns_orchestrator', u'tstamp': 1404392631642, u'type': u'info'}

      [2014-07-03 06:04:05,849] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u"Haven't heard from a higher priority node or a master, so I'm taking over.", u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:51.435Z', u'module': u'mb_master', u'tstamp': 1404392631435, u'type': u'info'}

      [2014-07-03 06:04:05,849] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 3, u'text': u'Node ns_1@10.3.3.199 joined cluster', u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:41.841Z', u'module': u'ns_cluster', u'tstamp': 1404392621841, u'type': u'info'}

      [2014-07-03 06:04:05,850] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@10.3.3.199\'. Version: "3.0.0-918-rel-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2014-07-03T06:03:41.716Z', u'module': u'menelaus_sup', u'tstamp': 1404392621716, u'type': u'info'}

      [2014-07-03 06:04:05,850] - [rest_client:2000] ERROR -

      {u'node': u'ns_1@10.3.3.199', u'code': 0, u'text': u"Current master is older and I'll try to takeover", u'shortText': u'message', u'serverTime': u'2014-07-03T06:03:40.911Z', u'module': u'mb_master', u'tstamp': 1404392620911, u'type': u'warning'}

      ERROR

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              alkondratenko Aleksey Kondratenko (Inactive)
              sangharsh Sangharsh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty