Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-20034

[Windows] Rebalance failed - Compactor for database terminated unexpectedly

    XMLWordPrintable

Details

    Description

      Rebalance fails because - Compactor for database terminated unexpectedly

      [2016-06-27 01:02:26,689] - [rest_client:1065] INFO - adding remote node @172.23.105.58:8091 to this cluster @172.23.105.239:8091
      [2016-06-27 01:02:32,112] - [rest_client:1283] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.239%2Cns_1%40172.23.105.58
      [2016-06-27 01:02:32,118] - [rest_client:1287] INFO - rebalance operation started
      [2016-06-27 01:02:32,124] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-06-27 01:02:42,151] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-06-27 01:02:52,172] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-06-27 01:03:02,193] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-06-27 01:03:12,213] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-06-27 01:03:22,234] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
      [2016-06-27 01:03:32,255] - [rest_client:1401] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
      [2016-06-27 01:03:32,272] - [rest_client:2601] INFO - Latest logs from UI on 172.23.105.239:
      [2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.239', u'code': 2, u'text': u"Rebalance exited with reason {not_all_nodes_are_ready_yet,\n                                 ['ns_1@172.23.105.58']}\n", u'shortText': u'message', u'serverTime': u'2016-06-27T01:03:32.273Z', u'module': u'ns_orchestrator', u'tstamp': 1467014612273, u'type': u'info'}
      [2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.58', u'code': 0, u'text': u'Compactor for database `default` (pid [{type,database},\n                                       {important,true},\n                                       {name,<<"default">>},\n                                       {fa,\n                                        {#Fun<compaction_new_daemon.4.92023696>,\n                                         [<<"default">>,\n                                          {config,\n                                           {30,18446744073709551616},\n                                           {30,18446744073709551616},\n                                           undefined,false,false,\n                                           {daemon_config,30,131072,20971520}},\n                                          false,\n                                          {[{type,bucket}]}]}}]) terminated unexpectedly: {{bad_return_value,\n                                                                                            {stop,\n                                                                                             {error,\n                                                                                              couldnt_connect_to_memcached}}},\n                                                                                           {gen_server,\n                                                                                            call,\n                                                                                            [{\'ns_memcached-default\',\n                                                                                              \'ns_1@172.23.105.58\'},\n                                                                                             {raw_stats,\n                                                                                              <<"diskinfo">>,\n                                                                                              #Fun<compaction_new_daemon.18.92023696>,\n                                                                                              {<<"0">>,\n                                                                                               <<"0">>}},\n                                                                                             180000]}}', u'shortText': u'message', u'serverTime': u'2016-06-27T01:03:02.170Z', u'module': u'compaction_new_daemon', u'tstamp': 1467014582170, u'type': u'warning'}
      [2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.239', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2016-06-27T01:02:32.271Z', u'module': u'ns_rebalancer', u'tstamp': 1467014552271, u'type': u'info'}
      [2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.239', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.239','ns_1@172.23.105.58'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-06-27T01:02:32.237Z', u'module': u'ns_orchestrator', u'tstamp': 1467014552237, u'type': u'info'}
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-20034
          # Subject Branch Project Status CR V

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty