[System Test][XDCR] error observed in longevity - ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque xxx timed out chan 0xxxx

7.1.1-3043

Test:
-test tests/integration/neo/test_neo_magma_milestone4.yml -scope tests/integration/neo/scope_neo_magma.yml
Scale 3
Iteration 2

This test ran for 2 days and in this duration the following error occurred more than 5 times:

172.23.108.136 : xdcr
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log:2022-05-10T03:01:22.073-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 4237557760 timed out chan 0xc002c21a48

Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.104.249.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.105.0.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.105.38.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.105.39.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.106.37.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.107.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.132.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.139.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.140.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.141.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.142.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.143.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.144.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.145.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.146.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652181957/collectinfo-2022-05-10T112601-ns_1%40172.23.108.148.zip

172.23.105.38 : xdcr
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log.1.gz:2022-05-10T06:07:20.839-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 2438791168 timed out chan 0xc000388310

172.23.107.236 : xdcr
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log:2022-05-10T06:07:17.179-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 508166144 timed out chan 0xc00521c718

Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.104.249.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.105.0.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.105.38.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.105.39.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.106.37.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.107.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.107.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.132.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.138.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.139.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.140.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.141.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.142.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.143.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.144.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.145.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.146.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652193188/collectinfo-2022-05-10T143309-ns_1%40172.23.108.148.zip

172.23.107.25 : xdcr
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log.1.gz:2022-05-10T07:28:23.305-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 3742498816 timed out chan 0xc0048024f0

172.23.108.139 : xdcr
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log.1.gz:2022-05-10T07:27:42.139-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given type VBMasterCheck opaque 991100928 timed out chan 0xc005d06870

Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.104.249.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.105.0.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.105.38.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.105.39.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.106.37.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.107.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.107.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.132.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.138.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.139.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.140.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.142.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.143.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.144.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.145.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.146.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1652199358/collectinfo-2022-05-10T161559-ns_1%40172.23.108.148.zip