Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28782

[FTS Scorch] scorch index closing leads to cbft exit

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • FTS Sprint Mar-30-2018

    Description

      Build : 5.5.0-2211

      Test : ./testrunner -i /tmp/testexec.23344.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,GROUP=P1,index_type=scorch -t fts.moving_topology_fts.MovingTopFTS.rebalance_out_during_index_building,items=30000,cluster=D,F,F,index_replicas=1,standard_buckets=2,sasl_buckets=2,GROUP=P1

      Job : centos-fts_moving-topology-P1-scorch

      Rebalance out fails when an index is being built with the following error:

      [2018-03-18 10:50:27,454] - [rest_client:1569] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
      		 
      [2018-03-18 10:50:27,484] - [rest_client:3102] INFO - Latest logs from UI on 172.23.105.90:
      		 
      [2018-03-18 10:50:27,484] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n                                 {lost_connection,shutdown}}', u'shortText': u'message', u'serverTime': u'2018-03-18T10:50:24.019Z', u'module': u'ns_orchestrator', u'tstamp': 1521395424019, u'type': u'critical'}
      		 
      [2018-03-18 10:50:27,484] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.106.10', u'code': 0, u'text': u'Service \'fts\' exited with status 1. Restarting. Messages:\n2018-03-18T10:50:19.919-07:00 [INFO] cbdatasource: server: 172.23.105.90:11210, uprOpenName: fts:standard_bucket_2_index_1_132a08cd82f16a02_13aa53f3-41e2aaf6, worker, looping beg, vbucketState: "running" (has 171 vbuckets), 0-170\n2018-03-18T10:50:20.961-07:00 [INFO] pindex_bleve: batchWorker stopped \n2018-03-18T10:50:21.740-07:00 [INFO] cbdatasource: server: 172.23.105.90:11210, uprOpenName: fts:standard_bucket_2_index_1_132a08cd82f16a02_6ddbfb54-1285b624, worker, looping beg, vbucketState: "running" (has 169 vbuckets), 855-1023\n2018-03-18T10:50:21.765-07:00 [INFO] cbdatasource: server: 172.23.105.90:11210, uprOpenName: fts:standard_bucket_2_index_1_132a08cd82f16a02_54820232-21e9ebf, worker, looping beg, vbucketState: "running" (has 171 vbuckets), 342-512\n2018-03-18T10:50:21.807-07:00 [FATA] scorch AsyncError, treating this as fatal, err: got err persisting snapshot: scorch closed -- main.initBleveOptions.func1() at init_bleve.go:85\n[goport(/opt/couchbase/bin/cbft)] 2018/03/18 10:50:21 child process exited with status 1\n', u'shortText': u'message', u'serverTime': u'2018-03-18T10:50:21.937Z', u'module': u'ns_log', u'tstamp': 1521395421937, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,485] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:28.458Z', u'module': u'ns_vbucket_mover', u'tstamp': 1521395368458, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,485] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:28.320Z', u'module': u'ns_rebalancer', u'tstamp': 1521395368320, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,485] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Bucket "sasl_bucket_1" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:28.287Z', u'module': u'ns_vbucket_mover', u'tstamp': 1521395368287, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,486] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Started rebalancing bucket sasl_bucket_1', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:28.127Z', u'module': u'ns_rebalancer', u'tstamp': 1521395368127, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,486] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Bucket "sasl_bucket_2" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:28.098Z', u'module': u'ns_vbucket_mover', u'tstamp': 1521395368098, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,486] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Started rebalancing bucket sasl_bucket_2', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:27.926Z', u'module': u'ns_rebalancer', u'tstamp': 1521395367926, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,486] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Bucket "standard_bucket_1" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:27.903Z', u'module': u'ns_vbucket_mover', u'tstamp': 1521395367903, u'type': u'info'}
      		 
      [2018-03-18 10:50:27,487] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.105.90', u'code': 0, u'text': u'Started rebalancing bucket standard_bucket_1', u'shortText': u'message', u'serverTime': u'2018-03-18T10:49:27.729Z', u'module': u'ns_rebalancer', u'tstamp': 1521395367729, u'type': u'info'}

      Attachments

        For Gerrit Dashboard: MB-28782
        # Subject Branch Project Status CR V
        91519,3 bump up bleve SHA for master/vulcan manifests master manifest Status: MERGED +2 +1

        Activity

          People

            abhinav Abhi Dangeti
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty