Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48836

rest_create_index: error creating index: unable to create agent, err: agent setup failed, err: ambiguous timeout

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1444
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

      • Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
      • Create 1 bucket, 100 scopes and 10 collections in each scopes
      • Create 2500 GSI indexes ( 5 on each collection)
      • Load documents on some of the collections
      • Created 1000 indexes: one index (1 partition) on each collection

      See some index creation failed with below:

      {"log":"2021-10-08 21:36:30,388 - ftsindexmanager - INFO - Content = b'{\"error\":\"rest_create_index: error creating index: bucket_bucket1_idx_aevfa_7-country, err: manager_api: failed to fetch sourceUUID for sourceName: bucket1, sourceType: gocbcore, err: gocbcore_utils: CBSourceUUIDLookUp, unable to create agent, err: agent setup failed, err: ambiguous timeout | {\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{},\\\\\"Message\\\\\":\\\\\"ambiguous timeout\\\\\"}},\\\\\"OperationID\\\\\":\\\\\"CMD_HELLO\\\\\",\\\\\"Opaque\\\\\":\\\\\"0x1\\\\\",\\\\\"TimeObserved\\\\\":1051924,\\\\\"RetryReasons\\\\\":null,\\\\\"RetryAttempts\\\\\":0,\\\\\"LastDispatchedTo\\\\\":\\\\\"172.31.50.69:11207\\\\\",\\\\\"LastDispatchedFrom\\\\\":\\\\\"172.31.50.113:55566\\\\\",\\\\\"LastConnectionID\\\\\":\\\\\"d913b373c0be1bf3/35409392f8cc46e9\\\\\"}\",\"request\":{\"name\":\"bucket_bucket1_idx_aevfa_7-country\",\"params\":{\"doc_config\":{\"docid_prefix_delim\":\"\",\"docid_regexp\":\"\",\"mode\":\"scope.collection.type_field\",\"type_field\":\"type\"},\"mapping\":{\"default_analyzer\":\"keyword\",\"default_datetime_parser\":\"dateTimeOptional\",\"default_field\":\"_all\",\"default_mapping\":{\"dynamic\":true,\"enabled\":false},\"default_type\":\"_default\",\"docvalues_dynamic\":true,\"index_dynamic\":true,\"store_dynamic\":false,\"type_field\":\"_type\",\"types\":{\"scope_80.coll_2\":{\"dynamic\":false,\"enabled\":true,\"properties\":{\"country\":{\"dynamic\":false,\"enabled\":true,\"fields\":[{\"index\":true,\"name\":\"country\",\"type\":\"text\"}]}}}}}},\"planParams\":{\"indexPartitions\":1,\"maxPartitionsPerPIndex\":512},\"sourceName\":\"bucket1\",\"sourceType\":\"gocbcore\",\"store\":{\"indexType\":\"scorch\"},\"type\":\"fulltext-index\"},\"status\":\"fail\"}\\n' \n","stream":"stderr","time":"2021-10-09T04:36:30.389262935Z"}
       
       
       
      {"log":"2021-10-08 16:28:02,860 - ftsindexmanager - INFO - Content = b'{\"error\":\"rest_create_index: error creating index: bucket_bucket1_idx_mknyh_1-country, err: manager_api: failed to fetch sourceUUID for sourceName: bucket1, sourceType: gocbcore, err: gocbcore_utils: CBSourceUUIDLookUp, unable to create agent, err: agent setup failed, err: unambiguous timeout | {\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{},\\\\\"Message\\\\\":\\\\\"unambiguous timeout\\\\\"}},\\\\\"OperationID\\\\\":\\\\\"WaitUntilReady\\\\\",\\\\\"Opaque\\\\\":\\\\\"\\\\\",\\\\\"TimeObserved\\\\\":10000539695,\\\\\"RetryReasons\\\\\":[\\\\\"NOT_READY\\\\\"],\\\\\"RetryAttempts\\\\\":15,\\\\\"LastDispatchedTo\\\\\":\\\\\"\\\\\",\\\\\"LastDispatchedFrom\\\\\":\\\\\"\\\\\",\\\\\"LastConnectionID\\\\\":\\\\\"\\\\\"}\",\"request\":{\"name\":\"bucket_bucket1_idx_mknyh_1-country\",\"params\":{\"doc_config\":{\"docid_prefix_delim\":\"\",\"docid_regexp\":\"\",\"mode\":\"scope.collection.type_field\",\"type_field\":\"type\"},\"mapping\":{\"default_analyzer\":\"keyword\",\"default_datetime_parser\":\"dateTimeOptional\",\"default_field\":\"_all\",\"default_mapping\":{\"dynamic\":true,\"enabled\":false},\"default_type\":\"_default\",\"docvalues_dynamic\":true,\"index_dynamic\":true,\"store_dynamic\":false,\"type_field\":\"_type\",\"types\":{\"scope_99.coll_8\":{\"dynamic\":false,\"enabled\":true,\"properties\":{\"country\":{\"dynamic\":false,\"enabled\":true,\"fields\":[{\"index\":true,\"name\":\"country\",\"type\":\"text\"}]}}}}}},\"planParams\":{\"indexPartitions\":1,\"maxPartitionsPerPIndex\":512},\"sourceName\":\"bucket1\",\"sourceType\":\"gocbcore\",\"store\":{\"indexType\":\"scorch\"},\"type\":\"fulltext-index\"},\"status\":\"fail\"}\\n' \n","stream":"stderr","time":"2021-10-08T23:28:02.860600657Z"}
       
       
      {"log":"2021-10-08 15:08:22,863 - ftsindexmanager - INFO - Content = b'{\"error\":\"rest_create_index: error creating index: bucket_bucket1_idx_gzxhs_6-country, err: manager_api: failed to fetch sourceUUID for sourceName: bucket1, sourceType: gocbcore, err: gocbcore_utils: CBSourceUUIDLookUp, unable to create agent, err: agent setup failed, err: ambiguous timeout | {\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{},\\\\\"Message\\\\\":\\\\\"ambiguous timeout\\\\\"}},\\\\\"OperationID\\\\\":\\\\\"CMD_HELLO\\\\\",\\\\\"Opaque\\\\\":\\\\\"0x1\\\\\",\\\\\"TimeObserved\\\\\":36418,\\\\\"RetryReasons\\\\\":null,\\\\\"RetryAttempts\\\\\":0,\\\\\"LastDispatchedTo\\\\\":\\\\\"172.31.50.61:11207\\\\\",\\\\\"LastDispatchedFrom\\\\\":\\\\\"172.31.50.113:40490\\\\\",\\\\\"LastConnectionID\\\\\":\\\\\"4c18b4c2427ac106/23c48faa356c4812\\\\\"}\",\"request\":{\"name\":\"bucket_bucket1_idx_gzxhs_6-country\",\"params\":{\"doc_config\":{\"docid_prefix_delim\":\"\",\"docid_regexp\":\"\",\"mode\":\"scope.collection.type_field\",\"type_field\":\"type\"},\"mapping\":{\"default_analyzer\":\"keyword\",\"default_datetime_parser\":\"dateTimeOptional\",\"default_field\":\"_all\",\"default_mapping\":{\"dynamic\":true,\"enabled\":false},\"default_type\":\"_default\",\"docvalues_dynamic\":true,\"index_dynamic\":true,\"store_dynamic\":false,\"type_field\":\"_type\",\"types\":{\"scope_91.coll_3\":{\"dynamic\":false,\"enabled\":true,\"properties\":{\"country\":{\"dynamic\":false,\"enabled\":true,\"fields\":[{\"index\":true,\"name\":\"country\",\"type\":\"text\"}]}}}}}},\"planParams\":{\"indexPartitions\":1,\"maxPartitionsPerPIndex\":512},\"sourceName\":\"bucket1\",\"sourceType\":\"gocbcore\",\"store\":{\"indexType\":\"scorch\"},\"type\":\"fulltext-index\"},\"status\":\"fail\"}\\n' \n","stream":"stderr","time":"2021-10-08T22:08:22.863538873Z"}
       
       
      {"log":"2021-10-08 13:31:27,171 - ftsindexmanager - INFO - Content = b'{\"error\":\"rest_create_index: error creating index: bucket_bucket1_idx_knsql-country, err: manager_api: failed to fetch sourceUUID for sourceName: bucket1, sourceType: gocbcore, err: gocbcore_utils: CBSourceUUIDLookUp, unable to create agent, err: agent setup failed, err: ambiguous timeout | {\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{},\\\\\"Message\\\\\":\\\\\"ambiguous timeout\\\\\"}},\\\\\"OperationID\\\\\":\\\\\"CMD_HELLO\\\\\",\\\\\"Opaque\\\\\":\\\\\"0x1\\\\\",\\\\\"TimeObserved\\\\\":96880,\\\\\"RetryReasons\\\\\":null,\\\\\"RetryAttempts\\\\\":0,\\\\\"LastDispatchedTo\\\\\":\\\\\"172.31.50.113:11207\\\\\",\\\\\"LastDispatchedFrom\\\\\":\\\\\"172.31.50.113:55114\\\\\",\\\\\"LastConnectionID\\\\\":\\\\\"5d152640ded48481/979cba32623279a2\\\\\"}\",\"request\":{\"name\":\"bucket_bucket1_idx_knsql-country\",\"params\":{\"doc_config\":{\"docid_prefix_delim\":\"\",\"docid_regexp\":\"\",\"mode\":\"scope.collection.type_field\",\"type_field\":\"type\"},\"mapping\":{\"default_analyzer\":\"keyword\",\"default_datetime_parser\":\"dateTimeOptional\",\"default_field\":\"_all\",\"default_mapping\":{\"dynamic\":true,\"enabled\":false},\"default_type\":\"_default\",\"docvalues_dynamic\":true,\"index_dynamic\":true,\"store_dynamic\":false,\"type_field\":\"_type\",\"types\":{\"scope_78.coll_9\":{\"dynamic\":false,\"enabled\":true,\"properties\":{\"country\":{\"dynamic\":false,\"enabled\":true,\"fields\":[{\"index\":true,\"name\":\"country\",\"type\":\"text\"}]}}}}}},\"planParams\":{\"indexPartitions\":1,\"maxPartitionsPerPIndex\":512},\"sourceName\":\"bucket1\",\"sourceType\":\"gocbcore\",\"store\":{\"indexType\":\"scorch\"},\"type\":\"fulltext-index\"},\"status\":\"fail\"}\\n' \n","stream":"stderr","time":"2021-10-08T20:31:27.172190419Z"}
       
       
      {"log":"2021-10-08 12:55:58,796 - ftsindexmanager - INFO - Content = b'{\"error\":\"rest_create_index: error creating index: bucket_bucket1_idx_gqpge_1-country, err: manager_api: failed to fetch sourceUUID for sourceName: bucket1, sourceType: gocbcore, err: gocbcore_utils: CBSourceUUIDLookUp, unable to create agent, err: agent setup failed, err: ambiguous timeout | {\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{\\\\\"InnerError\\\\\":{},\\\\\"Message\\\\\":\\\\\"ambiguous timeout\\\\\"}},\\\\\"OperationID\\\\\":\\\\\"CMD_SELECTBUCKET\\\\\",\\\\\"Opaque\\\\\":\\\\\"0x5\\\\\",\\\\\"TimeObserved\\\\\":39859,\\\\\"RetryReasons\\\\\":null,\\\\\"RetryAttempts\\\\\":0,\\\\\"LastDispatchedTo\\\\\":\\\\\"172.31.50.113:11207\\\\\",\\\\\"LastDispatchedFrom\\\\\":\\\\\"172.31.50.113:50298\\\\\",\\\\\"LastConnectionID\\\\\":\\\\\"399d8c19bb8dab9b/66854d7b13969c91\\\\\"}\",\"request\":{\"name\":\"bucket_bucket1_idx_gqpge_1-country\",\"params\":{\"doc_config\":{\"docid_prefix_delim\":\"\",\"docid_regexp\":\"\",\"mode\":\"scope.collection.type_field\",\"type_field\":\"type\"},\"mapping\":{\"default_analyzer\":\"keyword\",\"default_datetime_parser\":\"dateTimeOptional\",\"default_field\":\"_all\",\"default_mapping\":{\"dynamic\":true,\"enabled\":false},\"default_type\":\"_default\",\"docvalues_dynamic\":true,\"index_dynamic\":true,\"store_dynamic\":false,\"type_field\":\"_type\",\"types\":{\"scope_73.coll_8\":{\"dynamic\":false,\"enabled\":true,\"properties\":{\"country\":{\"dynamic\":false,\"enabled\":true,\"fields\":[{\"index\":true,\"name\":\"country\",\"type\":\"text\"}]}}}}}},\"planParams\":{\"indexPartitions\":1,\"maxPartitionsPerPIndex\":512},\"sourceName\":\"bucket1\",\"sourceType\":\"gocbcore\",\"store\":{\"indexType\":\"scorch\"},\"type\":\"fulltext-index\"},\"status\":\"fail\"}\\n' \n","stream":"stderr","time":"2021-10-08T19:55:58.797178808Z"}
      
      

      And also seeing "CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout" messages in fts logs.

      ns_server.fts.log:2021-10-08T22:25:08.577+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:53.275+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:54.267+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:56.593+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:56.602+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:56.604+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:56.606+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:56.627+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:57.697+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:57.728+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:57.784+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:57.889+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:26:57.895+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:27:02.883+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:27:03.214+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:29:51.758+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:27.443+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:28.635+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:28.797+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:28.863+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:29.222+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:32.244+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:32.246+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:35.454+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:36.645+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:36.817+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:36.875+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      ns_server.fts.log:2021-10-08T22:30:37.266+00:00 [WARN] (GOCBCORE) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:646
      

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/amb/collectinfo-2021-10-09T072138-ns_1%40172.31.50.113.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/amb/collectinfo-2021-10-09T072138-ns_1%40172.31.50.61.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/amb/collectinfo-2021-10-09T072138-ns_1%40172.31.50.69.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty