Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27589

[FTS] Dgm-compaction tests stuck > 3 mins at indexing phase in recent builds

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Critical
    • 5.5.0
    • 5.5.0
    • fts
    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Build
      5.5.0-1686

      Testcase
      /data/workspace/cen006-p0-fts-vset03-00-dgm-compaction-mossstore/logs/testrunner-18-Jan-14_01-53-30/test_7

      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,level_compaction=False,GROUP=MSTORE -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D,F,D+F,dgm_run=1,active_resident_ratio=10,eviction_policy=fullEviction,moss_compact_threshold=20,GROUP=MSTORE

      Is a regression introduced between 5.5.0-1607 and 5.5.0-1632. Might be related to MB-27588.

      Steps
      1. Create a [D+F, F, D+F] - 3 node cluster
      2. create a bucket with full eviction policy and load until active resident ratio is 10%
      3. create default index with moss compaction threshold=20
      4. check if all docs get indexed.

      [2018-01-14 07:40:11,743] - [fts_base:876] INFO - Creating fulltext-index default_index_1 on 172.23.105.216
      		 
      [2018-01-14 07:40:11,744] - [rest_client:2607] INFO - {"params": {"store": {"mossStoreOptions": {"CompactionPercentage": 20}, "kvStoreName": "mossStore"}}, "name": "default_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""}
      		 
      [2018-01-14 07:40:11,804] - [rest_client:2614] INFO - Index default_index_1 created
      		 
      [2018-01-14 07:40:11,804] - [fts_base:3481] INFO - Validating index distribution for default_index_1 ...
      		 
      [2018-01-14 07:40:12,009] - [fts_base:3499] INFO - Validated: Number of PIndexes = 6
      		 
      [2018-01-14 07:40:12,033] - [fts_base:3511] INFO - Validated: Every pIndex serves 171 partitions or lesser
      		 
      [2018-01-14 07:40:12,034] - [fts_base:3529] INFO - Validated: pIndexes are distributed across [u'e8174385616c0c7bf4f862b21ae16dc3', u'f2955605dbd83950324d505ff36efb5a', u'8a6e226f619a1ca86339a2457adbb1d3'] 
      [2018-01-14 07:40:12,034] - [fts_base:3535] INFO - Expecting num of partitions in each node in range 170-512
      		 
      [2018-01-14 07:40:12,034] - [fts_base:3550] INFO - Validated: Node e8174385616c0c7bf4f862b21ae16dc3 houses 2 pindexes which serve 342 partitions
      		 
      [2018-01-14 07:40:12,035] - [fts_base:3550] INFO - Validated: Node f2955605dbd83950324d505ff36efb5a houses 2 pindexes which serve 342 partitions
      		 
      [2018-01-14 07:40:12,035] - [fts_base:3550] INFO - Validated: Node 8a6e226f619a1ca86339a2457adbb1d3 houses 2 pindexes which serve 340 partitions
      		 
      [2018-01-14 07:40:12,286] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:40:12,358] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 0
      		 
      [2018-01-14 07:40:18,464] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:40:18,536] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 21833
      		 
      [2018-01-14 07:40:24,790] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:40:24,869] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 44912
      		 
      [2018-01-14 07:40:31,234] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:40:31,315] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 68781
      		 
      :
      		 
      :
      		 
      [2018-01-14 07:53:30,248] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:53:30,358] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3013706
      		 
      [2018-01-14 07:53:36,498] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:53:36,562] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3024655
      		 
      [2018-01-14 07:53:42,779] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:53:42,901] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3035544
      		 
      [2018-01-14 07:53:49,031] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:53:49,128] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3048349
      		 
      [2018-01-14 07:53:55,267] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:53:55,375] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3058099
      		 
      [2018-01-14 07:54:01,479] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:01,596] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3069728
      		 
      [2018-01-14 07:54:07,833] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:07,935] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3080803
      		 
      [2018-01-14 07:54:14,318] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:14,413] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3090481
      		 
      [2018-01-14 07:54:20,536] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:20,641] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3099343
      		 
      [2018-01-14 07:54:26,811] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:26,917] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3106850
      		 
      [2018-01-14 07:54:33,089] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:33,197] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3115944
      		 
      [2018-01-14 07:54:39,366] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:39,452] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3125687
      		 
      [2018-01-14 07:54:45,752] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:45,852] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3135654
      		 
      [2018-01-14 07:54:52,037] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:52,144] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3143590
      		 
      [2018-01-14 07:54:58,259] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:54:58,363] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3151169
      		 
      [2018-01-14 07:55:04,600] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:04,728] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3158403
      		 
      [2018-01-14 07:55:10,847] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:10,957] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3167961
      		 
      [2018-01-14 07:55:17,025] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:17,088] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3173334
      		 
      [2018-01-14 07:55:23,225] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:23,316] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3176957
      		 
      [2018-01-14 07:55:29,390] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:29,457] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3180795
      		 
      [2018-01-14 07:55:35,608] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:35,723] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3182127
      		 
      [2018-01-14 07:55:41,860] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:41,931] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3192978
      		 
      [2018-01-14 07:55:47,999] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:48,058] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3201814
      		 
      [2018-01-14 07:55:54,164] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:55:54,256] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209162
      		 
      [2018-01-14 07:56:00,332] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:00,397] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:06,469] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:06,533] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:12,607] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:12,676] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:18,745] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:18,808] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:24,872] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:24,939] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:31,005] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:31,074] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:37,145] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:37,215] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:43,280] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:43,346] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:49,415] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:49,479] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:56:55,553] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:56:55,620] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:01,692] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:01,759] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:07,833] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:07,903] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:13,977] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:14,042] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:20,114] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:20,188] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:26,262] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:26,326] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:32,396] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:32,459] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:38,534] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:38,601] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:44,670] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:44,738] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:50,812] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:50,874] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:57:56,941] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:57:57,010] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:58:03,078] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:58:03,152] - [fts_base:3400] INFO - Docs in bucket = 3566519, docs in FTS index 'default_index_1': 3209514
      		 
      [2018-01-14 07:58:09,225] - [rest_client:1875] INFO - http://172.23.106.120:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-01-14 07:58:09,293] - [fts_base:3673] INFO - Docs in index default_index_1=3209514, bucket docs=3566519
      		 
      FAIL

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty