Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7511

rebalance failed during online upgrade - Rebalance exited with reason {{badmatch,[]}, [{ns_vbucket_mover, '-spawn_workers/

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.0.1
    • 2.0.1
    • installer, ns_server
    • Security Level: Public
    • None

    Description

      http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-upgrade/51/consoleFull
      ./testrunner -i /tmp/upgrade.ini get-logs=False,upgrade_version=2.0.1-120-rel,initial_vbuckets=64 -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade_and_rebalance,initial_version=2.0.0-1978-rel,nodes_init=4,num_stoped_nodes=1,nodes_in=1,nodes_out=1

      steps:
      1. cluster 2.0.0-1978-rel ns_1%4010.3.3.11%2Cns_1%4010.3.3.16%2Cns_1%4010.3.3.14%2Cns_1%4010.3.3.13
      2. start rebalance with ejectedNodes=ns_1%4010.3.3.13
      3.during rebalance upgrade 10.3.3.16 on 2.0.1-120-rel

      result: [2013-01-09 04:28:20,650] - [newupgradetests:139] INFO - rebalance failed as expected
      4.wait while upgrade 10.3.3.16 completed
      [2013-01-09 04:29:47,536] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.3.16:8091
      [2013-01-09 04:29:47,570] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:48,591] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:49,610] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:50,623] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:51,643] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:52,661] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:53,673] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:54,685] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:55,696] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:56,709] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:57,725] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:58,736] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:59,748] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:30:00,761] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:30:01,773] - [cluster_helper:86] INFO - ns_server @ 10.3.3.16:8091 is running
      5. and retry rebalance
      [2013-01-09 04:30:02,774] - [rest_client:788] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.13&user=Administrator&knownNodes=ns_1%4010.3.3.11%2Cns_1%4010.3.3.16%2Cns_1%4010.3.3.14%2Cns_1%4010.3.3.13
      [2013-01-09 04:30:02,784] - [rest_client:792] INFO - rebalance operation started
      [2013-01-09 04:30:02,797] - [rest_client:881] INFO - rebalance percentage : 0 %
      [2013-01-09 04:30:12,817] - [rest_client:866] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed

      result:

      2013-01-09 04:30:06.976 ns_orchestrator:2:info:message(ns_1@10.3.3.16) - Rebalance exited with reason {{badmatch,[]},
      [

      {ns_vbucket_mover, '-spawn_workers/1-lc$^0/1-0-',2}

      ,

      {ns_vbucket_mover,spawn_workers,1}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Aliaksey Artamonau Aliaksey Artamonau (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty