Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52277

[Backport MB-44158 to 6.6.6] [System Test] Seeing errors "DDLServiceMgr: Failed to start metadata provider" several times in the test

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-4390
      Test : -test tests/2i/cheshirecat/test_idx_steady_state_cheshire_cat_coll_crud.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat.yml
      Scale : 2
      Iteration : 2nd

      After running the steady state test for 6+ hrs, on all indexer nodes, around the same time, the following errors were seen -

      2021-02-06 20:09:06,698 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.121.165 ***
      2021-02-06 20:09:06,698 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.720-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.
      2021-02-06 20:09:10,940 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.96.30 ***
      2021-02-06 20:09:10,940 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.810-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.
      2021-02-06 20:09:14,246 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.77 ***
      2021-02-06 20:09:14,247 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:52.042-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.
      2021-02-06 20:09:17,742 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.82 ***
      2021-02-06 20:09:17,742 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.975-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.
      2021-02-06 20:09:21,065 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.83 ***
      2021-02-06 20:09:21,065 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.974-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.
      

      Didnt find any functional impact due to these errors, but if there were any DDL requests coming through during this timeframe, i think they would error out.

      Later in the test, there were several other instances of these types of errors across all nodes. This was the first occurence.

      Indexer nodes = 172.23.121.165, 172.23.96.30, 172.23.97.77, 172.23.97.82, 172.23.97.83
      Query nodes = 172.23.104.21, 172.23.104.23

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-52277
          # Subject Branch Project Status CR V

          Activity

            Changes that are needed for this are below.

            https://review.couchbase.org/c/indexing/+/153496 - MB-44158 : Skip fetching or storing stats in MetadataProvider
            Stats need not be fetched at watchers when the correspoding metadataprovider does not have statsNotifyCh

            https://review.couchbase.org/c/indexing/+/153672 - MB-44158 : Optmize getServiceMap to start watcher faster
            Make getServiceMap faster so that watcher can be initialized faster

            sai.teja Sai Krishna Teja added a comment - Changes that are needed for this are below. https://review.couchbase.org/c/indexing/+/153496 - MB-44158 : Skip fetching or storing stats in MetadataProvider Stats need not be fetched at watchers when the correspoding metadataprovider does not have statsNotifyCh https://review.couchbase.org/c/indexing/+/153672 - MB-44158 : Optmize getServiceMap to start watcher faster Make getServiceMap faster so that watcher can be initialized faster

            Might have to decide if the above are needed in 6.6.6
            The above issues were seen in system test

            CC Deepkaran Salooja, Varun Velamuri 

            sai.teja Sai Krishna Teja added a comment - Might have to decide if the above are needed in 6.6.6 The above issues were seen in system test CC Deepkaran Salooja , Varun Velamuri  

            Build couchbase-server-6.6.6-10557 contains indexing commit d1c9af1 with commit message:
            MB-52277: Optimise getServiceMap to start watcher faster

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.6-10557 contains indexing commit d1c9af1 with commit message: MB-52277 : Optimise getServiceMap to start watcher faster
            pavan.pb Pavan PB added a comment -

            System test on 6.6.6-10573 hasn't thrown any errors in its 2 day run so far. Closing this out.

            pavan.pb Pavan PB added a comment - System test on 6.6.6-10573 hasn't thrown any errors in its 2 day run so far. Closing this out.

            People

              pavan.pb Pavan PB
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty