Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31166

Rebalance failure after hard failover on 3 Nodes(NC+NC+CC)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.0.0
    • 6.0.0
    • analytics
    • Enterprise Edition 6.0.0 build 1603

    Description

      Setup: 5 Node cluster, 1 KV and 4 CBAS

      1. Load 100k documents on KV
      2. Create dataset and connect bucket
      3. hard fail over 3 nodes(CC + 1st NC + 2nd NC)
      4. Rebalance
      5. Rebalance fails

      Note: No secondary index created

      [2018-09-05 21:50:51,605] - [rest_client:1441] INFO - rebalance percentage : 50.00 %
      		 
      [2018-09-05 21:51:01,618] - [rest_client:1425] ERROR - {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed
      		 
      [2018-09-05 21:51:01,690] - [rest_client:2553] INFO - Latest logs from UI on 172.23.121.239:
      		 
      [2018-09-05 21:51:01,690] - [rest_client:2554] ERROR - {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.121.239', u'tstamp': 1536209451980L, u'shortText': u'message', u'serverTime': u'2018-09-05T21:50:51.980Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {rebalance_failed,\n                               {service_error,\n                                <<"wait for condition on http://172.23.107.56:9111/analytics/cluster timed out">>}}}'}
      		 
      [2018-09-05 21:51:01,691] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.120.160', u'tstamp': 1536209151975L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.975Z', u'text': u"Node 'ns_1@172.23.120.160' saw that node 'ns_1@172.23.109.224' went down. Details: [{nodedown_reason,\n                                                                                     connection_closed}]"}
      		 
      [2018-09-05 21:51:01,691] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.239', u'tstamp': 1536209151972L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.972Z', u'text': u"Node 'ns_1@172.23.121.239' saw that node 'ns_1@172.23.109.224' went down. Details: [{nodedown_reason,\n                                                                                     connection_closed}]"}
      		 
      [2018-09-05 21:51:01,693] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.239', u'tstamp': 1536209151642L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.642Z', u'text': u"Node 'ns_1@172.23.121.239' saw that node 'ns_1@172.23.104.76' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]"}
      		 
      [2018-09-05 21:51:01,693] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.120.160', u'tstamp': 1536209151642L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.642Z', u'text': u"Node 'ns_1@172.23.120.160' saw that node 'ns_1@172.23.104.76' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]"}
      		 
      [2018-09-05 21:51:01,693] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.109.224', u'tstamp': 1536209151641L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.641Z', u'text': u"Node 'ns_1@172.23.109.224' saw that node 'ns_1@172.23.104.76' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]"}
      		 
      [2018-09-05 21:51:01,694] - [rest_client:2554] ERROR - {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.121.239', u'tstamp': 1536209151580L, u'shortText': u'message', u'serverTime': u'2018-09-05T21:45:51.580Z', u'text': u'Bucket "default" rebalance appears to be swap rebalance'}
      		 
      [2018-09-05 21:51:01,694] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.239', u'tstamp': 1536209151564L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.564Z', u'text': u"Node 'ns_1@172.23.121.239' saw that node 'ns_1@172.23.107.56' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]"}
      		 
      [2018-09-05 21:51:01,694] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.109.224', u'tstamp': 1536209151556L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.556Z', u'text': u"Node 'ns_1@172.23.109.224' saw that node 'ns_1@172.23.107.56' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]"}
      		 
      [2018-09-05 21:51:01,694] - [rest_client:2554] ERROR - {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.120.160', u'tstamp': 1536209151556L, u'shortText': u'node down', u'serverTime': u'2018-09-05T21:45:51.556Z', u'text': u"Node 'ns_1@172.23.120.160' saw that node 'ns_1@172.23.107.56' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]"}
      		 
      ERROR

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            tanzeem.ahmed Tanzeem Ahmed (Inactive)
            tanzeem.ahmed Tanzeem Ahmed (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty