Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11029

Rebalance exited with reason {buckets_cleanup_failed if data folder contains some other 'error,eacces' files

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • 3.0
    • 3.0
    • ns_server
    • Security Level: Public
    • None
    • Untriaged
    • Unknown

    Description

      on my vms data path set as:
      "hdd":[{"path":"/tmp","index_path":"/tmp",

      I know that the best practice to have a data folder in isolation, but
      1) an attempt to delete all the files(not relating to CB) is also not true.
      2) whether to rebalance failed if a file was created in this folder with not CB user

      2014-05-03 00:13:52 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.4.146%2Cns_1%4010.3.4.144%2Cns_1%4010.3.4.148%2Cns_1%4010.3.4.149%2Cns_1%4010.3.4.145%2Cns_1%4010.3.4.147
      2014-05-03 00:13:52 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
      2014-05-03 00:13:52 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0 %
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      2014-05-03 00:14:02 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 10.3.4.144:
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 2, u'text': u"Rebalance exited with reason

      {buckets_cleanup_failed,\n ['ns_1@10.3.4.146','ns_1@10.3.4.144',\n 'ns_1@10.3.4.148','ns_1@10.3.4.149',\n 'ns_1@10.3.4.145','ns_1@10.3.4.147']}

      \n", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.720Z', u'module': u'ns_orchestrator', u'tstamp': 1399100884720, u'type': u'info'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 0, u'text': u"Failed to cleanup old buckets on node 'ns_1@10.3.4.146':

      {error,eacces}", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.720Z', u'module': u'ns_rebalancer', u'tstamp': 1399100884720, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 0, u'text': u"Failed to cleanup old buckets on node 'ns_1@10.3.4.144': {error,eacces}

      ", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.720Z', u'module': u'ns_rebalancer', u'tstamp': 1399100884720, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 0, u'text': u"Failed to cleanup old buckets on node 'ns_1@10.3.4.148':

      {error,eacces}", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.719Z', u'module': u'ns_rebalancer', u'tstamp': 1399100884719, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 0, u'text': u"Failed to cleanup old buckets on node 'ns_1@10.3.4.149': {error,eacces}

      ", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.719Z', u'module': u'ns_rebalancer', u'tstamp': 1399100884719, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 0, u'text': u"Failed to cleanup old buckets on node 'ns_1@10.3.4.145':

      {error,eacces}", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.718Z', u'module': u'ns_rebalancer', u'tstamp': 1399100884718, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.144', u'code': 0, u'text': u"Failed to cleanup old buckets on node 'ns_1@10.3.4.147': {error,eacces}

      ", u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.717Z', u'module': u'ns_rebalancer', u'tstamp': 1399100884717, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.149', u'code': 0, u'text': u'Unable to rm -rf bucket database directory atop.d\n

      {error,eacces}', u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.709Z', u'module': u'ns_storage_conf', u'tstamp': 1399100884709, u'type': u'critical'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.149', u'code': 0, u'text': u'Deleting old data files of bucket "atop.d"', u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.704Z', u'module': u'ns_storage_conf', u'tstamp': 1399100884704, u'type': u'info'}
      2014-05-03 00:14:02 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@10.3.4.147', u'code': 0, u'text': u'Unable to rm -rf bucket database directory atop.d\n{error,eacces}

      ', u'shortText': u'message', u'serverTime': u'2014-05-03T00:08:04.689Z', u'module': u'ns_storage_conf', u'tstamp': 1399100884689, u'type': u'critical'}
      [('./testrunner', 331, '<module>', 'result = unittest.TextTestRunner(verbosity=2).run(suite)'), ('/usr/lib64/python2.6/unittest.py', 752, 'run', 'test(result)'), ('/usr/lib64/python2.6/unittest.py', 463, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.6/unittest.py', 459, 'run', 'test(result)'), ('/usr/lib64/python2.6/unittest.py', 299, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.6/unittest.py', 278, 'run', 'testMethod()'), ('pytests/rebalance/rebalancein.py', 189, 'incremental_rebalance_in_with_ops', 'task.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]

      please note, I had not previously seen it in the same environment

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty