Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24086

[FTS] Windows: Indexing in dgm is at least 7x slower

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 5.0.0
    • 5.0.0
    • fts
    • Untriaged
    • Yes

    Description

      Build
      5.0.0-2703

      Testcase
      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D:F:F,GROUP=MSTORE -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D+F,F,F,dgm_run=1,active_resident_ratio=50,GROUP=MSTORE

      Steps
      1. In a D+F,F,F cluster, load to default bucket until we reach <= 50% active resident ratio.
      2. Create a default index.

      Windows vm config: 3 vms, 8 cores, 8GB RAM, 200GB ssd
      CentOS vm config: 3 vms, 4 cores, 8GB RAM, 187GB ssd

      Multiple tests are timing out(after 2 mins of no index count change) due to slowness during indexing. Same tests on CentOS pass though.

      Also noting that this is a regression, the last I ran this suite was in late Aug 2016 (before I went on leave). From logs, I see "[2016-08-26 16:10:16,312] - [fts_base:2602] INFO - FTS indexed 661000 docs in 1.83 mins". This was against 4.7.0-1021.

      On Windows - 7 mins into indexing, only 260k docs indexed with periodic ingestion(indicating mossStore compaction) -

      [2017-04-24 14:49:44,930] - [fts_base:3513] INFO - Validated: Node 3db85e3fdc902a0aa1bbff67e07c590b houses 2 pindexes which serve 340 partitions
      		 
      [2017-04-24 14:49:44,931] - [fts_base:3513] INFO - Validated: Node 0b7c0f3e761a06d8563ab3001b9bd868 houses 2 pindexes which serve 342 partitions
      		 
      [2017-04-24 14:49:44,931] - [fts_base:3513] INFO - Validated: Node da4bbb81665ea2c21ccf7a45bbd03cdd houses 2 pindexes which serve 342 partitions
      		 
      [2017-04-24 14:49:45,050] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 23720
      		 
      [2017-04-24 14:49:51,180] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 78839
      		 
      [2017-04-24 14:49:57,263] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 105129
      		 
      [2017-04-24 14:50:03,359] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 130022
      		 
      [2017-04-24 14:50:09,424] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 140543
      		 
      [2017-04-24 14:50:15,507] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 140543
      		 
      [2017-04-24 14:50:21,585] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 145159
      		 
      [2017-04-24 14:50:27,674] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 158128
      		 
      [2017-04-24 14:50:33,749] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:50:39,824] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:50:45,900] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:50:51,975] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:50:58,049] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:04,127] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:10,202] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:16,277] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:22,349] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:28,419] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:34,489] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:40,557] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 162859
      		 
      [2017-04-24 14:51:46,626] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 178479
      		 
      [2017-04-24 14:51:52,695] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 178479
      		 
      [2017-04-24 14:51:58,765] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 178479
      		 
      [2017-04-24 14:52:04,839] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:10,910] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:16,981] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:23,051] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:29,128] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:35,205] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:41,264] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:47,333] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:53,402] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 194295
      		 
      [2017-04-24 14:52:59,472] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 198775
      		 
      [2017-04-24 14:53:05,539] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 204179
      		 
      [2017-04-24 14:53:11,603] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 204179
      		 
      [2017-04-24 14:53:17,663] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 204179
      		 
      [2017-04-24 14:53:23,723] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 208815
      		 
      [2017-04-24 14:53:29,785] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 220580
      		 
      [2017-04-24 14:53:35,845] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 221955
      		 
      [2017-04-24 14:53:41,908] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 222084
      		 
      [2017-04-24 14:53:47,973] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 222084
      		 
      [2017-04-24 14:53:54,038] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 222084
      		 
      [2017-04-24 14:54:00,102] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 222084
      		 
      [2017-04-24 14:54:06,171] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:12,241] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:18,310] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:24,383] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:30,456] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:36,532] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:42,606] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:48,683] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:54:54,756] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:00,830] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:06,899] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:12,969] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:19,036] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:25,102] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:31,163] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:37,225] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:43,285] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:49,343] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:55:55,399] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:56:01,463] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:56:07,529] - [fts_base:3376] INFO - Docs in bucket = 571000, docs in FTS index 'default_index_1': 259128
      		 
      [2017-04-24 14:56:13,603] - [fts_base:3604] INFO - Docs in index default_index_1=259128, bucket docs=571000
      		 
      FAIL

      Same test on CentOS - indexing 641k docs takes only 3 mins.

       
      		 
      [2017-04-21 11:37:38,611] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 12335
      		 
      [2017-04-21 11:37:44,864] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 33680
      		 
      [2017-04-21 11:37:51,082] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 52109
      		 
      [2017-04-21 11:37:57,280] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 69600
      		 
      [2017-04-21 11:38:03,467] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 92238
      		 
      [2017-04-21 11:38:09,777] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 122776
      		 
      [2017-04-21 11:38:16,218] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 151714
      		 
      [2017-04-21 11:38:22,472] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 180328
      		 
      [2017-04-21 11:38:28,745] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 205048
      		 
      [2017-04-21 11:38:35,054] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 235796
      		 
      [2017-04-21 11:38:41,499] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 267674
      		 
      [2017-04-21 11:38:47,898] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 299887
      		 
      [2017-04-21 11:38:54,355] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 334612
      		 
      [2017-04-21 11:39:00,810] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 369959
      		 
      [2017-04-21 11:39:07,161] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 404674
      		 
      [2017-04-21 11:39:13,512] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 433279
      		 
      [2017-04-21 11:39:19,839] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 464430
      		 
      [2017-04-21 11:39:26,185] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 497339
      		 
      [2017-04-21 11:39:32,496] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 526300
      		 
      [2017-04-21 11:39:38,689] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 554233
      		 
      [2017-04-21 11:39:45,049] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 576179
      		 
      [2017-04-21 11:39:51,233] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 594647
      		 
      [2017-04-21 11:39:57,464] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 609300
      		 
      [2017-04-21 11:40:03,761] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 618710
      		 
      [2017-04-21 11:40:10,002] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 628255
      		 
      [2017-04-21 11:40:16,095] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 636279
      		 
      [2017-04-21 11:40:22,203] - [fts_base:3376] INFO - Docs in bucket = 641000, docs in FTS index 'default_index_1': 641000

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-24456 Service 'fts' exited with status 1 - error found in centos plasma longevity

          • Test Blocker - QE-related tests can't proceed
          • Closed
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. MB-24714 [FTS] Windows: Indexing in dgm is 2-3x slower

          • Major - Major loss of function.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              steve Steve Yen
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty