Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50879

janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed,

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-2223 and 7.1.0-2254
      TestSuite: custom-map-rqg-scorch-score-none_5.5_P1
      Test: ./testrunner -i /tmp/testexec.67438.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,cluster=D+F,index_type=scorch,fts_quota=1000,score=none,EXCLUDE_GROUP=COLLECTIONS_ONLY,GROUP=BUCKETS,get-cbcollect-info=True -t fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping,items=1000,custom_map=True,num_custom_analyzers=1,compare_es=True,cm_id=11,num_queries=100,GROUP=BUCKETS,get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,cluster=D+F,index_type=scorch,fts_quota=1000,score=none,EXCLUDE_GROUP=COLLECTIONS_ONLY

      Most tests in these testsuite are failing to index documents after index creation. Some tests are passing. All tests in this suite are of same steps but with different kinds of indexes. one of the instance is here.

      Index creation in the above test:

      [2022-02-07 20:40:35,135] - [rest_client:1109] INFO - Making a rest request api=http://172.23.108.102:8094/api/index/custom_index verb=PUT params={"type": "fulltext-index", "name": "custom_index", "uuid": "", "params": {"mapping": {"default_mapping": {"enabled": false, "dynamic": true, "default_analyzer": ""}, "type_field": "type", "default_type": "_default", "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "analysis": {"analyzers": {"customAnalyzer1": {"char_filters": ["html"], "token_filters": ["length"], "tokenizer": "whitespace", "type": "custom"}}, "token_filters": {"back_edge_ngram": {"back": true, "max": 5, "min": 3, "type": "edge_ngram"}, "dict_compound_en": {"dict_token_map": "stop_en", "type": "dict_compound"}, "dict_compound_fr": {"dict_token_map": "articles_fr", "type": "dict_compound"}, "front_edge_ngram": {"back": false, "max": 5, "min": 3, "type": "edge_ngram"}, "keyword_marker": {"keywords_token_map": "stopwords", "type": "keyword_marker"}, "stopwords": {"stop_token_map": "stopwords", "type": "stop_tokens"}, "length": {"max": 5, "min": 3, "type": "length"}, "ngram": {"max": 5, "min": 3, "type": "ngram"}, "shingle": {"filler": "", "max": 5, "min": 2, "output_original": "false", "separator": "", "type": "shingle"}, "truncate": {"length": 10, "type": "truncate_token"}}, "token_maps": {"stopwords": {"tokens": ["i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "would", "should", "could", "ought", "i'm", "you're", "he's", "she's", "it's", "we're", "they're", "i've", "you've", "we've", "they've", "i'd", "you'd", "he'd", "she'd", "we'd", "they'd", "i'll", "you'll", "he'll", "she'll", "we'll", "they'll", "isn't", "aren't", "wasn't", "weren't", "hasn't", "haven't", "hadn't", "doesn't", "don't", "didn't", "won't", "wouldn't", "shan't", "shouldn't", "can't", "cannot", "couldn't", "mustn't", "let's", "that's", "who's", "what's", "here's", "there's", "when's", "where's", "why's", "how's", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very"], "type": "custom"}}, "char_filters": {"mapping": {"regexp": "[f]", "replace": "ph", "type": "regexp"}}, "tokenizers": {"alphanumeric": {"regexp": "[0-9a-zA-Z_]*", "type": "regexp"}}}, "types": {"emp": {"dynamic": false, "enabled": true, "fields": [], "properties": {"manages": {"dynamic": false, "enabled": true, "fields": [], "properties": {"team_size": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "team_size", "store": false, "type": "number", "analyzer": ""}]}, "reports": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "reports", "store": false, "type": "text", "analyzer": "customAnalyzer1"}]}}}, "name": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "name", "store": false, "type": "text", "analyzer": "customAnalyzer1"}]}, "join_date": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "join_date", "store": false, "type": "datetime", "analyzer": ""}]}}}}}, "store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}} client_cert=None verify=False
      		 
      [2022-02-07 20:40:35,232] - [rest_client:3475] INFO - Index custom_index created

      From the fts log:

      2022-02-07T20:40:35.145-08:00 [INFO] rest_create_index: create index request received for custom_index
      		 
      2022-02-07T20:40:35.218-08:00 [INFO] cfg_metakv: Set, key: indexDefs, cas: 1, split: false, nodeUUID: 040606d436cc2dfcb0fe0c19f8f3b24e
      		 
      2022-02-07T20:40:35.219-08:00 [INFO] cfg_metakv: Set path: /fts/cbgt/cfg/indexDefs
      		 
      2022-02-07T20:40:35.223-08:00 [INFO] cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/indexDefs, key: indexDefs, deletion: false
      		 
      2022-02-07T20:40:35.227-08:00 [INFO] ctl: kickIndexDefs, kind: ctl: mgr event
      		 
      2022-02-07T20:40:35.230-08:00 [INFO] ctl: cfgEvent, kind: indexDefs
      		 
      2022-02-07T20:40:35.231-08:00 [INFO] manager_api: index definition created, indexType: fulltext-index, indexName: custom_index, indexUUID: 550d62f7d114e311

      In the test, we see index has 0 docs, even after some tries:

      [2022-02-07 20:40:39,445] - [rest_client:1109] INFO - Making a rest request api=http://172.23.108.102:8094/api/index/custom_index/count verb=GET params= client_cert=None verify=False
      		 
      [2022-02-07 20:40:39,456] - [rest_client:2535] INFO - http://172.23.108.102:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2022-02-07 20:40:39,795] - [fts_base:4687] INFO - Docs in bucket = 0, docs in FTS index 'custom_index': 0, docs in ES index: 1000 
      [2022-02-07 20:40:45,825] - [rest_client:1109] INFO - Making a rest request api=http://172.23.108.102:8094/api/index/custom_index/count verb=GET params= client_cert=None verify=False
      		 
      [2022-02-07 20:40:45,835] - [rest_client:2535] INFO - http://172.23.108.102:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2022-02-07 20:40:46,187] - [fts_base:4687] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 0, docs in ES index: 1000 
      [2022-02-07 20:40:52,216] - [rest_client:1109] INFO - Making a rest request api=http://172.23.108.102:8094/api/index/custom_index/count verb=GET params= client_cert=None verify=False
      		 
      [2022-02-07 20:40:52,227] - [rest_client:2535] INFO - http://172.23.108.102:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2022-02-07 20:40:52,586] - [fts_base:4687] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 0, docs in ES index: 1000 
      [2022-02-07 20:40:58,614] - [rest_client:1109] INFO - Making a rest request api=http://172.23.108.102:8094/api/index/custom_index/count verb=GET params= client_cert=None verify=False
      		 
      [2022-02-07 20:40:58,623] - [rest_client:2535] INFO - http://172.23.108.102:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2022-02-07 20:40:58,964] - [fts_base:4687] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 0, docs in ES index: 1000 
      [2022-02-07 20:41:04,994] - [rest_client:1109] INFO - Making a rest request api=http://172.23.108.102:8094/api/index/custom_index/count verb=GET params= client_cert=None verify=False
      		 
      [2022-02-07 20:41:05,005] - [rest_client:2535] INFO - http://172.23.108.102:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2022-02-07 20:41:05,364] - [fts_base:4687] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 0, docs in ES index: 1000

      In fts log, we see this WARN:

      2022-02-07T20:40:35.305-08:00 [INFO] pindex_bleve: started runBatchWorker: 0 for pindex: /opt/couchbase/var/lib/couchbase/data/@fts/custom_index_550d62f7d114e311_4c1c5584.pindex
      		 
      2022-02-07T20:40:35.305-08:00 [INFO] pindex_bleve: started runBatchWorker: 2 for pindex: /opt/couchbase/var/lib/couchbase/data/@fts/custom_index_550d62f7d114e311_4c1c5584.pindex
      		 
      2022-02-07T20:40:35.305-08:00 [INFO] pindex_bleve: started runBatchWorker: 1 for pindex: /opt/couchbase/var/lib/couchbase/data/@fts/custom_index_550d62f7d114e311_4c1c5584.pindex
      		 
      2022-02-07T20:40:35.305-08:00 [INFO] pindex_bleve: started runBatchWorker: 3 for pindex: /opt/couchbase/var/lib/couchbase/data/@fts/custom_index_550d62f7d114e311_4c1c5584.pindex
      		 
      2022-02-07T20:40:35.305-08:00 [INFO] janitor: feeds to remove: 0
      		 
      2022-02-07T20:40:35.305-08:00 [INFO] janitor: feeds to add: 1
      		 
      2022-02-07T20:40:35.305-08:00 [INFO]   custom_index_550d62f7d114e311_4c1c5584
      		 
      2022-02-07T20:40:35.307-08:00 [WARN] janitor: JanitorOnce, err: janitor: JanitorOnce errors: 1, []string{"#0: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: custom_index_550d62f7d114e311_4c1c5584, server: http://127.0.0.1:8091, bucketName: default, indexName: custom_index, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: mismatched bucketUUID"} -- cbgt.(*Manager).JanitorLoop() at manager_janitor.go:93

      Logs:
      172.23.108.102-20220207-2049-diag.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty