Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15753

[windows] Rebalance failed when ejecting nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 4.0.0
    • 4.0.0
    • view-engine
    • Security Level: Public
    • 3645

    Description

      1. Create a cluster of 5 nodes
      .144 - n1ql + kv
      .145 - kv
      .191 - kv
      .40 - index
      .91 - index
      2. Check if the cluster is healthy
      3. Create 3 buckets and add 100800 items to each bucket
      4. create primary index using views and secondary index using gsi. Wait for all indexes to be built.
      5. Eject out of the KV nodes and rebalance.
      [2015-07-20 05:50:54,363] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.191&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.40%2Cns_1%40172.23.107.144%2Cns_1%40172.23.107.145%2Cns_1%40172.23.105.91
      Rebalance fails:
      [2015-07-20 05:50:54,376] - [rest_client:1170] INFO - rebalance operation started
      [2015-07-20 05:50:54,469] - [rest_client:1288] INFO - rebalance percentage : 0.00 %
      [2015-07-20 05:51:04,510] - [rest_client:1288] INFO - rebalance percentage : 1.56 %
      [2015-07-20 05:51:14,613] - [rest_client:1288] INFO - rebalance percentage : 3.31 %
      [2015-07-20 05:51:24,632] - [rest_client:1288] INFO - rebalance percentage : 4.18 %
      [2015-07-20 05:51:34,650] - [rest_client:1288] INFO - rebalance percentage : 4.18 %
      [2015-07-20 05:51:44,670] - [rest_client:1288] INFO - rebalance percentage : 5.00 %
      [2015-07-20 05:51:54,749] - [rest_client:1288] INFO - rebalance percentage : 6.96 %
      [2015-07-20 05:52:04,801] - [rest_client:1288] INFO - rebalance percentage : 8.53 %
      [2015-07-20 05:52:14,821] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
      [2015-07-20 05:52:24,840] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
      [2015-07-20 05:52:34,860] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
      [2015-07-20 05:52:44,878] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
      [2015-07-20 05:52:54,898] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
      [2015-07-20 05:53:04,940] - [rest_client:1271] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      [2015-07-20 05:53:04,974] - [rest_client:2195] INFO - Latest logs from UI on 172.23.107.144:
      [2015-07-20 05:53:04,974] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.27919.31>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.107.145',\n {'EXIT',\n error,{badrpc,nodedown,\n {gen_server,call,\n [

      {'janitor_agent-standard_bucket0',\n 'ns_1@172.23.107.145'}

      ,\n {if_rebalance,<0.19056.31>,\n {update_vbucket_state,362,replica,\n undefined,'ns_1@172.23.107.144'}},\n

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritam.sharma Ritam Sharma
            ritam.sharma Ritam Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty