Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Untriaged
-
1
-
No
Description
7.1.0-1677 with p2p disabled
Test:
-test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
Scale 3
Iteration 1
Following errors observed in .106.100 goxdcr logs:
2021-11-09T20:05:35.638-08:00 INFO GOXDCR.RuntimeCtx: d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4 successfully stopped service BackfillMgrSvc.
|
2021-11-09T20:05:35.638-08:00 INFO GOXDCR.StatsMgr: d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4-567737452 expvar=Stats for pipeline d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4-567737452 {"Backfill Old Progress": "Source nozzles have been closed", "Backfill Progress": "Pipeline has been stopped", "Errors": "[{\"time\":\"2021-11-09T20:05:25.030536071-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.21:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:05:07.940021281-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.5:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:04:51.148858282-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.21:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:04:34.30676171-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.5:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:04:17.446469141-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.70:11210_1:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:04:00.570547128-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.97.121:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:03:43.402213497-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.97.122:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:03:26.480499628-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.108.103:11210_1:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:03:09.60922229-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.20:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:02:52.746939256-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.21:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:02:35.812363461-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.20:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:02:18.926237945-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.97.119:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:02:02.165256985-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.5:11210_1:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:01:45.304130926-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.5:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:01:28.339666586-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.21:11210_1:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:01:11.923660761-08:00\",\"errMsg\":\"pipelineMgr.pipeline_factory.NewPipeline(d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4):Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets.\"},{\"time\":\"2021-11-09T20:01:01.005353943-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.5:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:00:43.610128097-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.104.5:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:00:26.397306396-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.97.119:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"},{\"time\":\"2021-11-09T20:00:09.196211077-08:00\",\"errMsg\":\"dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.108.103:11210_1:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: \"}]", "OldProgress": "Pipeline failed to start, err=map[dcp_d0851ce9e1015bf9cebb3f08822e56de/bucket4/bucket4_172.23.99.21:11210_0:MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: ]", "Overview": {"CurrentTime": 1636517135638198456, "add_docs_written": 0, "bandwidth_usage": 0, "changes_left": 0, "data_merged": 0, "data_replicated": 0, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "docs_checked": 0, "docs_cloned": 0, "docs_failed_cr_source": 0, "docs_filtered": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 0, "docs_received_from_dcp": 0, "docs_rep_queue": 0, "docs_unable_to_filter": 0, "docs_written": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_filtered": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "num_checkpoints": 0, "num_failedckpts": 0, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 0, "set_docs_written": 0, "set_failed_cr_source": 0, "set_filtered": 0, "set_received_from_dcp": 0, "size_rep_queue": 0, "target_docs_skipped": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 0, "wtavg_docs_latency": 0, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Async listeners have been stopped", "Status": "Pending"}
|
Shows up in UI as well
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636562312/collectinfo-2021-11-10T163836-ns_1%40172.23.99.25.zip
Attachments
Issue Links
- relates to
-
MB-48884 [System Test] Few cluster operations failed due to error - Internal server error
- Closed
For Gerrit Dashboard: MB-49480 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
166799,2 | MB-49480: If "select bucket" command failed with KEY_ENOENT, don't show it in UI. | master | goxdcr | Status: ABANDONED | 0 | 0 |
166884,3 | MB-49480: Make sure KvVbMapPool.Get() returns clean map when node list is empty. | master | goxdcr | Status: MERGED | +2 | +1 |