Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49285

Search: 1K indexes: Indexes deleted after mutations

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1566
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

      • Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
      • Create 1 bucket, 100 scopes and 10 collections in each scopes
      • Create 2500 GSI indexes ( 5 on each collection)
      • Load documents on some of the collections
      • Created 1000 indexes: one index (1 partition) on each collection
      • Run queries on each collection
      • Mutate the documents on each collection and wait for all the index to process mutation
      • Run queries on each collection
      • Delete all the indexes

      After Mutating the documents in the steps above, its seems indexes are deleted.
      We do see below error messages thought out the period of mutating the documents in all collections.

      ns_server.fts.log:2021-10-28T15:04:09.419-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:04:12.684-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:05.216-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:05.216-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:05.306-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:05.546-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:11.693-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:11.958-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:11.978-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:11.988-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:11.990-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:11.994-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:12.001-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:12.473-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:13.220-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:13.220-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:13.311-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:13.550-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.020-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.639-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.649-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.942-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.962-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.980-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:18.983-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:19.697-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:19.962-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:19.982-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:19.991-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:19.993-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:19.999-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:20.005-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:20.477-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:21.224-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:21.226-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:21.316-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T15:48:21.554-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T16:01:38.648-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T16:26:56.946-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T16:26:59.701-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T16:27:04.950-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T16:51:45.979-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T16:51:46.024-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T17:20:17.146-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T17:20:19.455-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T17:48:42.765-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-28T17:48:42.772-07:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- 
      -- 
      -- ns_server.fts.log:2021-10-29T01:16:03.446-07:00 [WARN] (GOCBCORE) Pipeline Client 0xc09f4060e0 failed to bootstrap: ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"CMD_HELLO","Opaque":"0x1","TimeObserved":55747,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"172.23.100.161:11207","LastDispatchedFrom":"172.23.100.161:36184","LastConnectionID":"9212096889d6813b/20565048443f74aa"} -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
       
      		 
      ns_server.fts.log:2021-10-29T01:16:18.423-07:00 [WARN] (GOCBCORE) Pipeline Client 0xc013fc6230 failed to bootstrap: ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"CMD_HELLO","Opaque":"0x1","TimeObserved":60366,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"172.23.100.161:11207","LastDispatchedFrom":"172.23.100.161:36274","LastConnectionID":"d75f174966befeae/839fe22989166c78"} -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
       
      		 
      ns_server.fts.log:2021-10-29T01:17:18.576-07:00 [WARN] (GOCBCORE) Pipeline Client 0xc1473f40e0 failed to bootstrap: ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"CMD_HELLO","Opaque":"0x1","TimeObserved":63319,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"172.23.100.162:11207","LastDispatchedFrom":"172.23.100.161:32930","LastConnectionID":"57e1d0388b939f8b/7a25c091ef383695"} -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:679
      		 
      ns_server.fts.log:2021-10-29T01:17:19.189-07:00 [WARN] janitor: JanitorOnce, err: janitor: JanitorOnce errors: 6, []string{"#0: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bucket_bucket1_idx_xhudh_9-country_543c89e6d6993738_4c1c5584, server: http://127.0.0.1:8091, bucketName: bucket1, indexName: bucket_bucket1_idx_xhudh_9-country, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: ambiguous timeout | {\"InnerError\":{\"InnerError\":{\"InnerError\":{},\"Message\":\"ambiguous timeout\"}},\"OperationID\":\"CMD_HELLO\",\"Opaque\":\"0x1\",\"TimeObserved\":30196,\"RetryReasons\":null,\"RetryAttempts\":0,\"LastDispatchedTo\":\"172.23.100.163:11207\",\"LastDispatchedFrom\":\"172.23.100.161:54768\",\"LastConnectionID\":\"82e2294094e33a2f/be366823a2fff9af\"}", "#1: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bucket_bucket1_idx_fyaej_6-country_3b6912b4e839d712_4c1c5584, server: http://127.0.0.1:8091, bucketName: bucket1, indexName: bucket_bucket1_idx_fyaej_6-country, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: ambiguous timeout | {\"InnerError\":{\"InnerError\":{\"InnerError\":{},\"Message\":\"ambiguous timeout\"}},\"OperationID\":\"CMD_HELLO\",\"Opaque\":\"0x1\",\"TimeObserved\":95172,\"RetryReasons\":null,\"RetryAttempts\":0,\"LastDispatchedTo\":\"172.23.100.161:11207\",\"LastDispatchedFrom\":\"172.23.100.161:36158\",\"LastConnectionID\":\"4ad25cf85c5d7ca4/1e8b29f5dfd64cf3\"}", "#2: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bucket_bucket1_idx_vtiwf_3-country_62cc04283b26406e_4c1c5584, server: http://127.0.0.1:8091, bucketName: bucket1, indexName: bucket_bucket1_idx_vtiwf_3-country, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: ambiguous timeout | {\"InnerError\":{\"InnerError\":{\"InnerError\":{},\"Message\":\"ambiguous timeout\"}},\"OperationID\":\"CMD_HELLO\",\"Opaque\":\"0x1\",\"TimeObserved\":55747,\"RetryReasons\":null,\"RetryAttempts\":0,\"LastDispatchedTo\":\"172.23.100.161:11207\",\"LastDispatchedFrom\":\"172.23.100.161:36184\",\"LastConnectionID\":\"9212096889d6813b/20565048443f74aa\"}", "#3: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bucket_bucket1_idx_oyzvc_4-country_61b7e99f31a982d6_4c1c5584, server: http://127.0.0.1:8091, bucketName: bucket1, indexName: bucket_bucket1_idx_oyzvc_4-country, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: ambiguous timeout | {\"InnerError\":{\"InnerError\":{\"InnerError\":{},\"Message\":\"ambiguous timeout\"}},\"OperationID\":\"CMD_HELLO\",\"Opaque\":\"0x1\",\"TimeObserved\":60366,\"RetryReasons\":null,\"RetryAttempts\":0,\"LastDispatchedTo\":\"172.23.100.161:11207\",\"LastDispatchedFrom\":\"172.23.100.161:36274\",\"LastConnectionID\":\"d75f174966befeae/839fe22989166c78\"}", "#4: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bucket_bucket1_idx_fqdta-country_3c324a22af35ae6b_4c1c5584, server: http://127.0.0.1:8091, bucketName: bucket1, indexName: bucket_bucket1_idx_fqdta-country, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: ambiguous timeout | {\"InnerError\":{\"InnerError\":{\"InnerError\":{},\"Message\":\"ambiguous timeout\"}},\"OperationID\":\"CMD_HELLO\",\"Opaque\":\"0x1\",\"TimeObserved\":64140,\"RetryReasons\":null,\"RetryAttempts\":0,\"LastDispatchedTo\":\"172.23.100.161:11207\",\"LastDispatchedFrom\":\"172.23.100.161:36334\",\"LastConnectionID\":\"20a841375a9b5646/bf8434d00d1b5898\"}", "#5: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bucket_bucket1_idx_afowd_9-country_256615801a0633a6_4c1c5584, server: http://127.0.0.1:8091, bucketName: bucket1, indexName: bucket_bucket1_idx_afowd_9-country, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: ambiguous timeout | {\"InnerError\":{\"InnerError\":{\"InnerError\":{},\"Message\":\"ambiguous timeout\"}},\"OperationID\":\"CMD_HELLO\",\"Opaque\":\"0x1\",\"TimeObserved\":63319,\"RetryReasons\":null,\"RetryAttempts\":0,\"LastDispatchedTo\":\"172.23.100.162:11207\",\"LastDispatchedFrom\":\"172.23.100.161:32930\",\"LastConnectionID\":\"57e1d0388b939f8b/7a25c091ef383695\"}"} -- cbgt.(*Manager).JanitorLoop() at manager_janitor.go:93

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-11-01T224403-ns_1%40172.23.100.161.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-11-01T224403-ns_1%40172.23.100.162.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2021-11-01T224403-ns_1%40172.23.100.163.zip

      Attachments

        For Gerrit Dashboard: MB-49285
        # Subject Branch Project Status CR V
        164865,5 MB-49285: Use cached agents to obtain collection manifest master cbgt Status: MERGED +2 +1

        Activity

          People

            abhinav Abhi Dangeti
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty