Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7511

rebalance failed during online upgrade - Rebalance exited with reason {{badmatch,[]}, [{ns_vbucket_mover, '-spawn_workers/

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: installer, ns_server
    • Security Level: Public
    • Labels:
      None

      Description

      http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-upgrade/51/consoleFull
      ./testrunner -i /tmp/upgrade.ini get-logs=False,upgrade_version=2.0.1-120-rel,initial_vbuckets=64 -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade_and_rebalance,initial_version=2.0.0-1978-rel,nodes_init=4,num_stoped_nodes=1,nodes_in=1,nodes_out=1

      steps:
      1. cluster 2.0.0-1978-rel ns_1%4010.3.3.11%2Cns_1%4010.3.3.16%2Cns_1%4010.3.3.14%2Cns_1%4010.3.3.13
      2. start rebalance with ejectedNodes=ns_1%4010.3.3.13
      3.during rebalance upgrade 10.3.3.16 on 2.0.1-120-rel

      result: [2013-01-09 04:28:20,650] - [newupgradetests:139] INFO - rebalance failed as expected
      4.wait while upgrade 10.3.3.16 completed
      [2013-01-09 04:29:47,536] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.3.16:8091
      [2013-01-09 04:29:47,570] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:48,591] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:49,610] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:50,623] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:51,643] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:52,661] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:53,673] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:54,685] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:55,696] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:56,709] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:57,725] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:58,736] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:29:59,748] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:30:00,761] - [rest_client:36] WARNING - server 10.3.3.16:8091 status is warmup
      [2013-01-09 04:30:01,773] - [cluster_helper:86] INFO - ns_server @ 10.3.3.16:8091 is running
      5. and retry rebalance
      [2013-01-09 04:30:02,774] - [rest_client:788] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.13&user=Administrator&knownNodes=ns_1%4010.3.3.11%2Cns_1%4010.3.3.16%2Cns_1%4010.3.3.14%2Cns_1%4010.3.3.13
      [2013-01-09 04:30:02,784] - [rest_client:792] INFO - rebalance operation started
      [2013-01-09 04:30:02,797] - [rest_client:881] INFO - rebalance percentage : 0 %
      [2013-01-09 04:30:12,817] - [rest_client:866] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed

      result:

      2013-01-09 04:30:06.976 ns_orchestrator:2:info:message(ns_1@10.3.3.16) - Rebalance exited with reason {{badmatch,[]},
      [

      {ns_vbucket_mover, '-spawn_workers/1-lc$^0/1-0-',2}

      ,

      {ns_vbucket_mover,spawn_workers,1}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}

      # Subject Project Status CR V
      For Gerrit Dashboard: &For+MB-7511=message:MB-7511

        Activity

        Hide
        andreibaranouski Andrei Baranouski added a comment -

        I see the same issue in some other tests of the same run, that are not related upgrade during rebalance case

        Show
        andreibaranouski Andrei Baranouski added a comment - I see the same issue in some other tests of the same run, that are not related upgrade during rebalance case
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-new-rebalance-mixed-cluster/15/console
        the same is in tests with mixed cluster: rebalance cases with 1.8.1 and 2.0.1 nodes

        Show
        andreibaranouski Andrei Baranouski added a comment - http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-new-rebalance-mixed-cluster/15/console the same is in tests with mixed cluster: rebalance cases with 1.8.1 and 2.0.1 nodes
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        What does "second time" means?

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - What does "second time" means?
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        first time rebalance is when upgrade has started( step 3)
        second - retry rebalance after upgrade has been completed
        but do not take this into account, the same thing happens in other cases

        Show
        andreibaranouski Andrei Baranouski added a comment - first time rebalance is when upgrade has started( step 3) second - retry rebalance after upgrade has been completed but do not take this into account, the same thing happens in other cases
        Show
        Aliaksey Artamonau Aliaksey Artamonau added a comment - http://review.couchbase.org/#/c/23965/
        Hide
        Aliaksey Artamonau Aliaksey Artamonau added a comment -

        merged

        Show
        Aliaksey Artamonau Aliaksey Artamonau added a comment - merged

          People

          • Assignee:
            Aliaksey Artamonau Aliaksey Artamonau
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes