Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52385

[Backport MB-46288 to 6.6.6] Optimize cinfo in newMetadataProvider

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      [Backport MB-46288 to 6.6.6] [System test upgrade] : [Error] : Error messages like "DDLServiceMgr:rebalanceDone(): Failed to initialize metadata provider" seen in indexer log during upgrade

      Scripts to Repro
      1. Run the 6.6.2 longevity test for 3 days.

      ./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      2. It had 27 nodes at the end of the test.
      3. Added 6 7.0.0(172.23.105.102,172.23.105.62,172.23.106.232,172.23.106.239,172.23.106.37, 172.23.106.246) nodes and rebalanced in and removed 6 node from 6.6.2(172.23.110.75,172.23.110.76,172.23.105.61,172.23.106.191,172.23.106.209,172.23.106.70)
      and rebalanced out.
      4. Failed over 6 nodes and graceful failover + recovery + rebalance.
      5. Now swap rebalance 6 nodes. 2 data + 2 index + 1 eventing + 1 analytics.
      6. Then again did bunch of rebalances(stop/start and rebalance out eventing) to workaround MB-46274 and MB-46246.

      At this point I saw error messages like the following on indexer.log

      172.23.105.62 : index

      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-05-13T03:46:33.130-07:00 [Error] DDLServiceMgr:rebalanceDone(): Failed to initialize metadata provider.  Error=DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds..
      

      cbcollect_info attached. This was not seen in upgrade during 6.6.2->9588 to 7.0.0-5033.
      Possibly a dup of MB-46284?  

      Attachments

        Issue Links

          Activity

            People

              hemant.rajput Hemant Rajput
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty