Loading...

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 6.0.0
Affects Version/s: 6.0.0
Component/s: fts
Labels:
- functional-test

Triage:
Triaged
Is this a Regression?:
Yes

Description

Build
6.0.0-1348

Is a regression between 6.0.0-1318 and 6.0.0-1348

Testcase
./testrunner -i /tmp/testexec.29296.ini -p get-cbcollect-info=True,disable_HTP=True,get-logs=False,stop-on-failure=False,cluster=D+F,index_type=scorch,fts_quota=512 -t fts.stable_topology_fts.StableTopFTS.test_doc_config,mode=docid_regexp_neg1,query=

{"query": "Delta"}

,expected_hits=0,consistency_level=at_plus,consistency_vectors="""{"travel-index": {"0": 123, "1": 444}}""",GROUP=P0

Steps
1. Load travel-sample bucket
2. Create default index
3. Update travel sample index

Looks very similar to ~~MB-29576~~ which was fixed in Vulcan and Alice.

[2018-07-23 16:14:10,348] - [fts_base:883] INFO - Creating fulltext-index travel-index on 172.23.104.100

[2018-07-23 16:14:10,348] - [rest_client:2699] INFO - {"params": {"store": {"mossStoreOptions": {}, "kvStoreName": "mossStore", "indexType": "scorch"}}, "name": "travel-index", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "travel-sample", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""}

[2018-07-23 16:14:10,399] - [rest_client:2706] INFO - Index travel-index created

[2018-07-23 16:14:10,399] - [fts_base:3570] INFO - Validating index distribution for travel-index ...

[2018-07-23 16:14:10,434] - [fts_base:3454] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ...

[2018-07-23 16:14:15,487] - [fts_base:3588] INFO - Validated: Number of PIndexes = 6

[2018-07-23 16:14:15,497] - [fts_base:3600] INFO - Validated: Every pIndex serves 171 partitions or lesser

[2018-07-23 16:14:15,497] - [fts_base:3625] INFO - Expecting num of partitions in each node in range 853-1024

[2018-07-23 16:14:15,497] - [fts_base:3640] INFO - Validated: Node d23fc282ab6259ef872c940c7fbcbddb houses 6 pindexes which serve 1024 partitions

[2018-07-23 16:14:15,497] - [fts_base:3454] INFO - sleep for 10 secs.  ...

[2018-07-23 16:14:25,548] - [fts_base:893] INFO - Updating fulltext-index travel-index on 172.23.104.100

[2018-07-23 16:14:25,548] - [rest_client:2713] INFO - {

   "params": {

      "doc_config": {

         "mode": "docid_regexp",

         "docid_regexp": "\\b[a-z]{8,}"

},

      "mapping": {

         "types": {

            "airport": {

               "dynamic": true,

               "display_order": 0,

               "default_analyzer": "en",

               "enabled": true

},

            "hotel": {

               "dynamic": true,

               "display_order": 0,

               "default_analyzer": "en",

               "enabled": true

},

         "default_mapping": {

            "enabled": false,

            "dynamic": false,

            "properties": {}

},

      "store": {

         "mossStoreOptions": {},

         "kvStoreName": "mossStore",

         "indexType": "scorch"

},

   "name": "travel-index",

   "planParams": {

      "numReplicas": 0,

      "maxPartitionsPerPIndex": 171

},

   "sourceName": "travel-sample",

   "sourceUUID": "",

   "sourceType": "couchbase",

   "type": "fulltext-index",

   "uuid": "4cc1a50298d850bd"

[2018-07-23 16:14:25,579] - [rest_client:2720] INFO - Index/alias travel-index updated

[2018-07-23 16:14:25,579] - [fts_base:3454] INFO - sleep for 15 secs.  ...

[2018-07-23 16:14:40,631] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:14:40,664] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:14:46,703] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:14:46,744] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:14:52,795] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:14:52,838] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:14:58,883] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:14:58,919] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:04,954] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:04,992] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:11,037] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:11,076] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:17,126] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:17,163] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:23,204] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:23,239] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:29,276] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:29,319] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:35,361] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:35,403] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:41,443] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:41,481] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:47,521] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:47,563] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:53,605] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:53,645] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:15:59,685] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:15:59,723] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:05,763] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:05,802] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:11,839] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:11,883] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:17,918] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:17,957] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:23,999] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:24,036] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:30,076] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:30,120] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:36,160] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:36,198] - [fts_base:3477] INFO - Docs in bucket = 31591, docs in FTS index 'travel-index': 0

[2018-07-23 16:16:42,237] - [rest_client:1925] INFO - http://172.23.104.100:8091/pools/default/buckets/travel-sample/stats?zoom=minute

[2018-07-23 16:16:42,279] - [fts_base:3763] INFO - Docs in index travel-index=0, bucket docs=31591

FAIL

On .100, fts.log reads

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: current memory usage: 529896975 over indexing quota 524288000

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: current memory usage: 529896975 over indexing quota 524288000

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: current memory usage: 529896975 over indexing quota 524288000

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: current memory usage: 529896975 over indexing quota 524288000

2018-07-23T16:18:45.109-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.110-07:00 [INFO] app_herder: persistence progress, waiting: 13

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: merging progress, waiting: 12

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: resuming upon memory reduction

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: current memory usage: 537840023 over indexing quota 524288000

2018-07-23T16:18:47.111-07:00 [INFO] app_herder: waiting for more memory to be available

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

172.23.104.100-20180723-1616-diag.zip
33.67 MB
24/Jul/18 11:56 AM
172.23.120.232-20180727-2338-diag.zip
21.03 MB
30/Jul/18 8:59 AM
172.23.120.232-20180727-2338-diag.zip
21.03 MB
30/Jul/18 8:59 AM

Issue Links

relates to

MB-28847 [FTS] Memory usage during index build overshoots RAM Quota

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[FTS] Indexing is blocked after update by app_herder

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty