Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56176

[System test ] : After system test upgrade from 7.1.4-3601 -> 7.2.0-5263 we keep getting continuous errors from indexer :- error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 0
    • No

    Description

      Steps to Repro
      1. Run neo longevity test for 3 days on 7.1.4-3601.

      ./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_neo_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.1.4-3601 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

      2. Upgrade this cluster to 7.2.0-5263 using a mix of online and offline upgrade strategies.
      3. Updated the bucket properties of the all the buckets post upgrade using the following rest api.

      [root@s20507w12r2 ~]# curl localhost:8091/pools/default/buckets/default  -u Administrator:password -X POST -d historyRetentionBytes=2147483648

      Started getting errors from indexer as shown below even after 3-4 days post upgrade continuously every 5 mins.

      172.23.104.249 : index

      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T22:46:02.662-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T22:51:02.761-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T22:56:02.814-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:01:02.776-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:06:02.656-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:11:02.707-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:16:02.844-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:21:02.777-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:26:02.767-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-03-27T23:31:02.707-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-03-27T20:16:02.742-07:00 [Error] DDLServiceMgr::cleanupTokens: newMetadataProvider returned error: DDLServiceMgr::cleanupTokens:: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 30 seconds.. Skipping cleanup.

      cbcollect_info attached. Not sure if this is a bug. Since I kept getting this error for almost 4 days post upgrade raising this bug.

      Attachments

        Issue Links

          Clones

          Bug - A problem which impairs or prevents the functions of the product. MB-55882 [BP to 7.2.0]Auth Error floods the logs for upgrade from 7.0.4 to 7.2.0

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              dhananjay.kshirsagar Dhananjay Kshirsagar
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty