Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41381

FTS: checkAndStoreStablePlanPIndexes: SIGSEGV: segmentation violation

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      Build: 6.6.0-7919

      Test failed: ./testrunner -i /tmp/testexec.4790.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,index_type=scorch,fts_quota=750 -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=1000,cluster=D,F,standard_buckets=3,sasl_buckets=3,index_per_bucket=3,update=True,expires=30,memory_only=True,GROUP=P0
      Job: mem-only-indexes-scorch

      Error:

      [user:info,2020-09-09T18:06:55.298-07:00,ns_1@172.23.121.66:<0.11991.0>:ns_log:crash_consumption_loop:69]Service 'fts' exited with status 2. Restarting. Messages:
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x97e5b4]
       
      goroutine 64 [running]:
      github.com/couchbase/cbgt.isStablePlan(0x0, 0xc003242900)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:917 +0x84
      github.com/couchbase/cbgt.(*Manager).checkAndStoreStablePlanPIndexes(0xc00005ae00, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:941 +0x56
      github.com/couchbase/cbgt.(*Manager).GetPlanPIndexes(0xc00005ae00, 0xc008e13f01, 0x0, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:821 +0x459
      github.com/couchbase/cbgt.(*Manager).StartCfg.func2(0xc00005ae00)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:281 +0xd7
      created by github.com/couchbase/cbgt.(*Manager).StartCfg
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:272 +0x7d
       
      [ns_server:info,2020-09-09T18:06:55.298-07:00,ns_1@172.23.121.66:rebalance_agent<0.12237.0>:rebalance_agent:handle_down:296]Rebalancer process <23202.25836.1> died (reason {service_rebalance_failed,
                                                       fts,
                                                       {agent_died,<0.12321.0>,
      

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2020-09-10T011001-ns_1%40172.23.121.65.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2020-09-10T011001-ns_1%40172.23.121.66.zip

      Steps in the test:

      • Setup 2 node cluster with data and search on each node
      • create 3 buckets with authType=sasl and replica = 1 : sasl_bucket_1, sasl_bucket_2, sasl_bucket_3
      • create 3 buckets with authType=none and replica = 1: standard_bucket_1, standard_bucket_2, standard_bucket_3
      • Create below indexes with "{"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}"

        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
         {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
         {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
         
        {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
        

      • Load 1000 docs to all buckets and let indexing complete
      • Delete all the indexes
      • delete the buckets
      • Rebalance out search node
      • Below error is seen while rebalance

        2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
        2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.121.66', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.121.65,ns_1@172.23.121.66'}
        2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
        2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
        2020-09-09 18:06:41 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %
        2020-09-09 18:06:51 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %
        2020-09-09 18:07:01 | ERROR | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
        2020-09-09 18:07:02 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.121.65:
        2020-09-09 18:07:02 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.121.66', u'code': 0, u'text': u"Service 'fts' exited with status 2. Restarting. Messages:\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x97e5b4]\n\ngoroutine 64 [running]:\ngithub.com/couchbase/cbgt.isStablePlan(0x0, 0xc003242900)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:917 +0x84\ngithub.com/couchbase/cbgt.(*Manager).checkAndStoreStablePlanPIndexes(0xc00005ae00, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:941 +0x56\ngithub.com/couchbase/cbgt.(*Manager).GetPlanPIndexes(0xc00005ae00, 0xc008e13f01, 0x0, 0x0, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:821 +0x459\ngithub.com/couchbase/cbgt.(*Manager).StartCfg.func2(0xc00005ae00)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:281 +0xd7\ncreated by github.com/couchbase/cbgt.(*Manager).StartCfg\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:272 +0x7d\n", u'shortText': u'message', u'serverTime': u'2020-09-09T18:06:55.298Z', u'module': u'ns_log', u'tstamp': 1599700015298, u'type': u'info'}
        

      • This issue is not seen with indexes without "{"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}"
      • This issue is not seen when we rebalance without deleting indexes and buckets

      Attachments

        For Gerrit Dashboard: MB-41381
        # Subject Branch Project Status CR V

        Activity

          Build couchbase-server-7.0.0-3069 contains cbgt commit 04c9f58 with commit message:
          MB-41381 - FTS: SIGSEGV: segmentation violation

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3069 contains cbgt commit 04c9f58 with commit message: MB-41381 - FTS: SIGSEGV: segmentation violation

          Build couchbase-server-6.6.1-9047 contains cbgt commit b6013cb with commit message:
          MB-41381 - FTS: SIGSEGV: segmentation violation

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9047 contains cbgt commit b6013cb with commit message: MB-41381 - FTS: SIGSEGV: segmentation violation

          Build couchbase-server-7.0.0-3079 contains cbgt commit b6013cb with commit message:
          MB-41381 - FTS: SIGSEGV: segmentation violation

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3079 contains cbgt commit b6013cb with commit message: MB-41381 - FTS: SIGSEGV: segmentation violation

          Build couchbase-server-6.6.0-7920 contains cbgt commit 4c8c151 with commit message:
          MB-41381: [BP] FTS: SIGSEGV: segmentation violation

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7920 contains cbgt commit 4c8c151 with commit message: MB-41381 : [BP] FTS: SIGSEGV: segmentation violation

          Verified this in 6.6.0-7920

          girish.benakappa Girish Benakappa added a comment - Verified this in 6.6.0-7920

          Closing this issue as this test passed in regression run of weekly build: 7.0.0-3216

          girish.benakappa Girish Benakappa added a comment - Closing this issue as this test passed in regression run of weekly build: 7.0.0-3216

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty