Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-4390
Test : -test tests/2i/cheshirecat/test_idx_steady_state_cheshire_cat_coll_crud.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat.yml
Scale : 2
Iteration : 2nd
After running the steady state test for 6+ hrs, on all indexer nodes, around the same time, the following errors were seen -
2021-02-06 20:09:06,698 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.121.165 ***
|
2021-02-06 20:09:06,698 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.720-08:00 [Error] DDLServiceMgr: Failed to start metadata provider. Internal Error = DDLServiceMgr: Failed to initialize metadata provider. Unable to connect to all indexer nodes within 5 seconds.
|
2021-02-06 20:09:10,940 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.96.30 ***
|
2021-02-06 20:09:10,940 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.810-08:00 [Error] DDLServiceMgr: Failed to start metadata provider. Internal Error = DDLServiceMgr: Failed to initialize metadata provider. Unable to connect to all indexer nodes within 5 seconds.
|
2021-02-06 20:09:14,246 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.77 ***
|
2021-02-06 20:09:14,247 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:52.042-08:00 [Error] DDLServiceMgr: Failed to start metadata provider. Internal Error = DDLServiceMgr: Failed to initialize metadata provider. Unable to connect to all indexer nodes within 5 seconds.
|
2021-02-06 20:09:17,742 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.82 ***
|
2021-02-06 20:09:17,742 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.975-08:00 [Error] DDLServiceMgr: Failed to start metadata provider. Internal Error = DDLServiceMgr: Failed to initialize metadata provider. Unable to connect to all indexer nodes within 5 seconds.
|
2021-02-06 20:09:21,065 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.83 ***
|
2021-02-06 20:09:21,065 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.974-08:00 [Error] DDLServiceMgr: Failed to start metadata provider. Internal Error = DDLServiceMgr: Failed to initialize metadata provider. Unable to connect to all indexer nodes within 5 seconds.
|
Didnt find any functional impact due to these errors, but if there were any DDL requests coming through during this timeframe, i think they would error out.
Later in the test, there were several other instances of these types of errors across all nodes. This was the first occurence.
Indexer nodes = 172.23.121.165, 172.23.96.30, 172.23.97.77, 172.23.97.82, 172.23.97.83
Query nodes = 172.23.104.21, 172.23.104.23
Attachments
Issue Links
- is a backport of
-
MB-44158 [System Test] Seeing errors "DDLServiceMgr: Failed to start metadata provider" several times in the test
-
- Closed
-
For Gerrit Dashboard: MB-52277 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
183505,1 | MB-52277: Skip fetching or storing stats in MetadataProvider | mad-hatter | indexing | Status: ABANDONED | 0 | 0 |
183506,5 | MB-52277: Optimise getServiceMap to start watcher faster | mad-hatter | indexing | Status: MERGED | +2 | +1 |