Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.6.0
-
Untriaged
-
1
-
Yes
Description
Build: 6.6.0-7919
Test failed: ./testrunner -i /tmp/testexec.4790.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,index_type=scorch,fts_quota=750 -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=1000,cluster=D,F,standard_buckets=3,sasl_buckets=3,index_per_bucket=3,update=True,expires=30,memory_only=True,GROUP=P0
Job: mem-only-indexes-scorch
Error:
[user:info,2020-09-09T18:06:55.298-07:00,ns_1@172.23.121.66:<0.11991.0>:ns_log:crash_consumption_loop:69]Service 'fts' exited with status 2. Restarting. Messages:
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x97e5b4]
|
|
goroutine 64 [running]:
|
github.com/couchbase/cbgt.isStablePlan(0x0, 0xc003242900)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:917 +0x84
|
github.com/couchbase/cbgt.(*Manager).checkAndStoreStablePlanPIndexes(0xc00005ae00, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:941 +0x56
|
github.com/couchbase/cbgt.(*Manager).GetPlanPIndexes(0xc00005ae00, 0xc008e13f01, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:821 +0x459
|
github.com/couchbase/cbgt.(*Manager).StartCfg.func2(0xc00005ae00)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:281 +0xd7
|
created by github.com/couchbase/cbgt.(*Manager).StartCfg
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:272 +0x7d
|
|
[ns_server:info,2020-09-09T18:06:55.298-07:00,ns_1@172.23.121.66:rebalance_agent<0.12237.0>:rebalance_agent:handle_down:296]Rebalancer process <23202.25836.1> died (reason {service_rebalance_failed,
|
fts,
|
{agent_died,<0.12321.0>,
|
Logs:
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2020-09-10T011001-ns_1%40172.23.121.65.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2020-09-10T011001-ns_1%40172.23.121.66.zip
Steps in the test:
- Setup 2 node cluster with data and search on each node
- create 3 buckets with authType=sasl and replica = 1 : sasl_bucket_1, sasl_bucket_2, sasl_bucket_3
- create 3 buckets with authType=none and replica = 1: standard_bucket_1, standard_bucket_2, standard_bucket_3
- Create below indexes with "{"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}"
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "default_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_1_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_2_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "sasl_bucket_3_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "sasl_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_1_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_1", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_2_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_2", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_2", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
{"params": {"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}, "name": "standard_bucket_3_index_3", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "standard_bucket_3", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {}, "uuid": ""}
- Load 1000 docs to all buckets and let indexing complete
- Delete all the indexes
- delete the buckets
- Rebalance out search node
- Below error is seen while rebalance
2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.121.66', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.121.65,ns_1@172.23.121.66'}
2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
2020-09-09 18:06:31 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2020-09-09 18:06:41 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %
2020-09-09 18:06:51 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %
2020-09-09 18:07:01 | ERROR | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
2020-09-09 18:07:02 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.121.65:
2020-09-09 18:07:02 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.121.66', u'code': 0, u'text': u"Service 'fts' exited with status 2. Restarting. Messages:\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x97e5b4]\n\ngoroutine 64 [running]:\ngithub.com/couchbase/cbgt.isStablePlan(0x0, 0xc003242900)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:917 +0x84\ngithub.com/couchbase/cbgt.(*Manager).checkAndStoreStablePlanPIndexes(0xc00005ae00, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:941 +0x56\ngithub.com/couchbase/cbgt.(*Manager).GetPlanPIndexes(0xc00005ae00, 0xc008e13f01, 0x0, 0x0, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:821 +0x459\ngithub.com/couchbase/cbgt.(*Manager).StartCfg.func2(0xc00005ae00)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:281 +0xd7\ncreated by github.com/couchbase/cbgt.(*Manager).StartCfg\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/manager.go:272 +0x7d\n", u'shortText': u'message', u'serverTime': u'2020-09-09T18:06:55.298Z', u'module': u'ns_log', u'tstamp': 1599700015298, u'type': u'info'}
- This issue is not seen with indexes without "{"store": {"mossLowerLevelStoreName": "", "kvStoreName": "moss"}}"
- This issue is not seen when we rebalance without deleting indexes and buckets
Attachments
For Gerrit Dashboard: MB-41381 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
135872,2 | MB-41381 - FTS: SIGSEGV: segmentation violation | master | cbgt | Status: MERGED | +2 | +1 |
135943,3 | MB-41381 - FTS: SIGSEGV: segmentation violation | mad-hatter | cbgt | Status: MERGED | +2 | +1 |
135987,1 | Merge remote-tracking branch 'origin/mad-hatter' | master | cbgt | Status: MERGED | +2 | +1 |
135994,2 | MB-41381: [BP] FTS: SIGSEGV: segmentation violation | 6.6.0 | cbgt | Status: MERGED | +2 | +1 |