FTS: checkAndStoreStablePlanPIndexes: SIGSEGV: segmentation violation

Build: 6.6.0-7919

Test failed: ./testrunner -i /tmp/testexec.4790.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,index_type=scorch,fts_quota=750 -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=1000,cluster=D,F,standard_buckets=3,sasl_buckets=3,index_per_bucket=3,update=True,expires=30,memory_only=True,GROUP=P0
Job: mem-only-indexes-scorch

Error:

[user:info,2020-09-09T18:06:55.298-07:00,ns_1@172.23.121.66:<0.11991.0>:ns_log:crash_consumption_loop:69]Service 'fts' exited with status 2. Restarting. Messages:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x97e5b4]

goroutine 64 [running]:

github.com/couchbase/cbgt.isStablePlan(0x0, 0xc003242900)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:917 +0x84

github.com/couchbase/cbgt.(*Manager).checkAndStoreStablePlanPIndexes(0xc00005ae00, 0x0)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:941 +0x56

github.com/couchbase/cbgt.(*Manager).GetPlanPIndexes(0xc00005ae00, 0xc008e13f01, 0x0, 0x0, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:821 +0x459

github.com/couchbase/cbgt.(*Manager).StartCfg.func2(0xc00005ae00)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:281 +0xd7

created by github.com/couchbase/cbgt.(*Manager).StartCfg

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:272 +0x7d

[ns_server:info,2020-09-09T18:06:55.298-07:00,ns_1@172.23.121.66:rebalance_agent<0.12237.0>:rebalance_agent:handle_down:296]Rebalancer process <23202.25836.1> died (reason {service_rebalance_failed,

                                                 fts,

                                                 {agent_died,<0.12321.0>,

Logs:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2020-09-10T011001-ns_1%40172.23.121.65.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2020-09-10T011001-ns_1%40172.23.121.66.zip

Steps in the test:

• Setup 2 node cluster with data and search on each node
• create 3 buckets with authType=sasl and replica = 1 : sasl_bucket_1, sasl_bucket_2, sasl_bucket_3
• create 3 buckets with authType=none and replica = 1: standard_bucket_1, standard_bucket_2, standard_bucket_3
• Create below indexes with "{"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}"

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

  {"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}

• Load 1000 docs to all buckets and let indexing complete
• Delete all the indexes
• delete the buckets
• Rebalance out search node
• Below error is seen while rebalance

  2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes

  2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.121.66', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.121.65,ns_1@172.23.121.66'}

  2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started

  2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %

  2020-09-09 18:06:41 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %

  2020-09-09 18:06:51 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %

  2020-09-09 18:07:01 | ERROR | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

  2020-09-09 18:07:02 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.121.65:

  2020-09-09 18:07:02 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.121.66', u'code': 0, u'text': u"Service 'fts' exited with status 2. Restarting. Messages:\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x97e5b4]\n\ngoroutine 64 [running]:\ngithub.com/couchbase/cbgt.isStablePlan(0x0, 0xc003242900)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:917 +0x84\ngithub.com/couchbase/cbgt.(*Manager).checkAndStoreStablePlanPIndexes(0xc00005ae00, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:941 +0x56\ngithub.com/couchbase/cbgt.(*Manager).GetPlanPIndexes(0xc00005ae00, 0xc008e13f01, 0x0, 0x0, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:821 +0x459\ngithub.com/couchbase/cbgt.(*Manager).StartCfg.func2(0xc00005ae00)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:281 +0xd7\ncreated by github.com/couchbase/cbgt.(*Manager).StartCfg\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:272 +0x7d\n", u'shortText': u'message', u'serverTime': u'2020-09-09T18:06:55.298Z', u'module': u'ns_log', u'tstamp': 1599700015298, u'type': u'info'}

• This issue is not seen with indexes without "{"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}"
• This issue is not seen when we rebalance without deleting indexes and buckets