Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-20135

[FTS] cbft continuously crashes trying to write into pindex files after index and bucket deletion

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Critical
    • 4.5.1, 5.0.0
    • 4.5.1, 5.0.0
    • fts
    • Untriaged
    • No

    Description

      Build
      4.7.0-857

      Testcase (new)
      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D+F:F:F,kvstore=forestdb,GROUP=DGM -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D+F,F,F,dgm_run=1,active_resident_ratio=10,standard_buckets=1,sasl_buckets=1,fdb_compact_interval=300,fdb_compact_threshold=5,GROUP=DGM

      Testrunner log - https://gist.github.com/arunapiravi/277b7b8a5a08bafd6eb26e56887ee6b1

      There were multiple crashes seen in forestdb leading to MB-20134. We then started teardown phase in which cbft continued to attempt to write into pindex files while the index and subsequently the buckets were deleted resulting in deletion of pindex files.

      Attaching cbcollect info.

      You can see from the testrunner log that by 19:54:54, all pindex files were deleted following index deletion rest calls.

      [2016-07-11 19:54:36,235] - [fts_base:648] INFO - Validated: all index files for default_index_1 deleted from disk
      [2016-07-11 19:54:42,375] - [fts_base:648] INFO - Validated: all index files for sasl_bucket_1_index_1 deleted from disk
      [2016-07-11 19:54:54,229] - [fts_base:648] INFO - Validated: all index files for standard_bucket_1_index_1 deleted from disk

      In .216's fts log, we see cbft trying to write into index files @ and after 19:55:17-

      [2016-07-11 19:55:58,900] - [rest_client:1401] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
      		 
      [2016-07-11 19:55:58,931] - [rest_client:2605] INFO - Latest logs from UI on 172.23.106.120:
      		 
      [2016-07-11 19:55:58,931] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:58.573-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:58.573-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:58 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:58.575Z', u'module': u'ns_log', u'tstamp': 1468292158575, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,931] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:53.474-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:53.474-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:53 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:53.477Z', u'module': u'ns_log', u'tstamp': 1468292153477, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.106.120', u'code': 2, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n                                 {linked_process_died,<12826.27565.3>,\n                                     {no_connection,"fts-service_api"}}}\n', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:50.158Z', u'module': u'ns_orchestrator', u'tstamp': 1468292150158, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:48.378-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:48.378-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:48 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:48.381Z', u'module': u'ns_log', u'tstamp': 1468292148381, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:43.276-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:43.277-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:43 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:43.279Z', u'module': u'ns_log', u'tstamp': 1468292143279, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:38.183-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:38.183-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:38 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:38.186Z', u'module': u'ns_log', u'tstamp': 1468292138186, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:33.087-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:33.087-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:33 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:33.090Z', u'module': u'ns_log', u'tstamp': 1468292133090, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:27.997-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:27.997-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:27 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:28.000Z', u'module': u'ns_log', u'tstamp': 1468292128000, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:22.904-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:22.904-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:22 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:22.907Z', u'module': u'ns_log', u'tstamp': 1468292122907, u'type': u'info'}
      		 
      [2016-07-11 19:55:58,932] - [rest_client:2606] ERROR - {u'node': u'ns_1@172.23.105.216', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages: 2016-07-11T19:55:17.807-07:00 [INFO] main: data dir: "/data/@fts"\n2016-07-11T19:55:17.807-07:00 [FATAL] main: could not write uuidPath: /data/@fts/cbft.uuid\n  Please check that your -data/-dataDir parameter ("/data/@fts")\n  is to a writable directory where cbft can persist data. -- main.main() at main.go:159\n[goport] 2016/07/11 19:55:17 /opt/couchbase/bin/cbft terminated: exit status 1', u'shortText': u'message', u'serverTime': u'2016-07-11T19:55:17.809Z', u'module': u'ns_log', u'tstamp': 1468292117809, u'type': u'info'}

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-18766 Core files created by go-processes for every crash fill up disk

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              steve Steve Yen
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty