Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27284

[FTS - Scorch toybuild] Indexing does not progress for >2 mins + some pindexes unavailable for querying, err:'snappy: corrupt input'

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • fts
    • Untriaged
    • Centos 64-bit
    • Unknown

    Description

      Build
      5.1.0-1004 (first Scorch toybuild)

      Testcase
      /data/workspace/centos-p0-fts-vset00-00-defmap-P0/logs/testrunner-17-Dec-14_15-25-08/test_2

      ./testrunner -i /tmp/testexec.17401.ini cluster=D+F,get-cbcollect-info=True,GROUP=P0 -t fts.stable_topology_fts.StableTopFTS.test_query_type,items=100000,num_queries=1000,query_types=match,cluster=D,F,F,compare_es=True,GROUP=P0

      This is probably the reason behind MB-27283.

       
      		 
      [2017-12-14 15:28:36,997] - [fts_base:872] INFO - Creating fulltext-index default_index on 172.23.105.54
      		 
      [2017-12-14 15:28:36,998] - [rest_client:2594] INFO - {"params": {"store": {"mossStoreOptions": {}, "kvStoreName": "mossStore"}}, "name": "default_index", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""}
      		 
      [2017-12-14 15:28:37,018] - [rest_client:2601] INFO - Index default_index created
      		 
      [2017-12-14 15:28:37,018] - [fts_base:3477] INFO - Validating index distribution for default_index ...
      		 
      [2017-12-14 15:28:37,083] - [fts_base:3495] INFO - Validated: Number of PIndexes = 6
      		 
      [2017-12-14 15:28:37,096] - [fts_base:3507] INFO - Validated: Every pIndex serves 171 partitions or lesser
      		 
      [2017-12-14 15:28:37,096] - [fts_base:3531] INFO - Expecting num of partitions in each node in range 853-1024
      		 
      [2017-12-14 15:28:37,097] - [fts_base:3546] INFO - Validated: Node cdea08c7e16471a288dc7ee9c4dea3f7 houses 6 pindexes which serve 1024 partitions
      		 
      [2017-12-14 15:28:37,133] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:28:37,170] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 0, docs in ES index: 100000 
      [2017-12-14 15:28:43,461] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:28:43,501] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 11259, docs in ES index: 100000 
      [2017-12-14 15:28:49,885] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:28:49,913] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 22862, docs in ES index: 100000 
      [2017-12-14 15:28:56,248] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:28:56,283] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 35816, docs in ES index: 100000 
      [2017-12-14 15:29:25,365] - [rest_client:808] ERROR - socket error while connecting to http://172.23.105.54:8094/api/index/default_index/count error [Errno 111] Connection refused 
      [2017-12-14 15:29:28,452] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:29:28,481] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:29:34,566] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:29:34,590] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:29:40,624] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:29:40,648] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:29:46,689] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:29:46,714] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:29:52,758] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:29:52,779] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:29:58,816] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:29:58,840] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:04,879] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:04,904] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:10,942] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:10,970] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:17,005] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:17,033] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:23,071] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:23,100] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:29,136] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:29,163] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:35,214] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:35,246] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:41,284] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:41,312] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:47,347] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:47,375] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:53,409] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:53,438] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:30:59,475] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:30:59,502] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:31:05,538] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:31:05,567] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:31:11,606] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:31:11,634] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:31:17,674] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:31:17,700] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:31:23,737] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:31:23,764] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:31:29,802] - [rest_client:1862] INFO - http://172.23.105.54:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2017-12-14 15:31:29,829] - [fts_base:3405] INFO - Docs in bucket = 100000, docs in FTS index 'default_index': 17420, docs in ES index: 100000 
      [2017-12-14 15:31:36,522] - [task:1284] INFO - ----------------------------------------------------- Query # 1 ----------------------------------------------------
      		 
      [2017-12-14 15:31:36,535] - [fts_base:1967] INFO - Running query {"indexName": "default_index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "manages.reports", "match": "Hanna Drucilla"}, "size": 10000000} on node: 172.23.105.54:
      		 
      [2017-12-14 15:31:36,574] - [task:1288] INFO - Status: {u'successful': 5, u'failed': 1, u'total': 6, u'errors': {u'default_index_6766913e2da1a762_aa574717': u'snappy: corrupt input'}}
      		 
      [2017-12-14 15:31:36,574] - [task:1305] INFO - FTS returned partial results...skipping ES validation
      		 
      [2017-12-14 15:31:36,575] - [task:1312] INFO - FTS hits for query: {"field": "manages.reports", "match": "Hanna Drucilla"} is 1287 (took 2.890638ms)
      		 
      [2017-12-14 15:31:37,447] - [task:1322] INFO - ES hits for query: {"query": {"match": {"manages.reports": "Hanna Drucilla"}}} on es_index is 9481 (took 204ms)

      Attaching logs from .54.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty