Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 6.6.6
Affects Version/s: Cheshire-Cat
Component/s: secondary-index
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.16.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.17.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.18.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.19.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.23.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.121.165.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.96.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.97.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.97.82.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.97.83.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.16.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.17.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.18.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.19.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.21.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.104.23.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.121.165.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.96.30.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.97.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.97.82.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1612671232/collectinfo-2021-02-07T041353-ns_1%40172.23.97.83.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.0.0-4390
Test : -test tests/2i/cheshirecat/test_idx_steady_state_cheshire_cat_coll_crud.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat.yml
Scale : 2
Iteration : 2nd

After running the steady state test for 6+ hrs, on all indexer nodes, around the same time, the following errors were seen -

2021-02-06 20:09:06,698 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.121.165 ***

2021-02-06 20:09:06,698 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.720-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.

2021-02-06 20:09:10,940 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.96.30 ***

2021-02-06 20:09:10,940 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.810-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.

2021-02-06 20:09:14,246 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.77 ***

2021-02-06 20:09:14,247 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:52.042-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.

2021-02-06 20:09:17,742 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.82 ***

2021-02-06 20:09:17,742 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.975-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.

2021-02-06 20:09:21,065 - systestmon - WARNING - *** 1 occurences of Failed to initialize metadata provider keyword found on 172.23.97.83 ***

2021-02-06 20:09:21,065 - systestmon - DEBUG - /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-02-06T19:54:51.974-08:00 [Error] DDLServiceMgr: Failed to start metadata provider.  Internal Error = DDLServiceMgr: Failed to initialize metadata provider.  Unable to connect to all indexer nodes within 5 seconds.

Didnt find any functional impact due to these errors, but if there were any DDL requests coming through during this timeframe, i think they would error out.

Later in the test, there were several other instances of these types of errors across all nodes. This was the first occurence.

Indexer nodes = 172.23.121.165, 172.23.96.30, 172.23.97.77, 172.23.97.82, 172.23.97.83
Query nodes = 172.23.104.21, 172.23.104.23

Attachments

Issue Links

is a backport of

MB-44158 [System Test] Seeing errors "DDLServiceMgr: Failed to start metadata provider" several times in the test

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Pavan PB

Reporter:: Amit Kulkarni

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Due:: 27/May/21

Created:: 24/May/22 6:39 AM

Updated:: 04/Jan/23 2:24 AM

Resolved:: 04/Dec/22 11:26 PM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

MB-52277: Skip fetching or storing stats in MetadataProvider: Gerrit Review:

MB-52277: Optimise getServiceMap to start watcher faster: Gerrit Review:

[Backport MB-44158 to 6.6.6] [System Test] Seeing errors "DDLServiceMgr: Failed to start metadata provider" several times in the test

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty