[System Test] batchGetMeta received fatal error and had to abort - error observed in longevity

7.1.0-1377

Test:
-test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
Scale 3
Iteration 1

172.23.106.100 : xdcr
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log.1.gz:2021-09-29T05:50:58.437-07:00 ERRO GOXDCR.XmemNozzle: xmem_8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4_172.23.105.168:11210_1 batchGetMeta received fatal error and had to abort. Expected 53 responses, got 0 responses. err=Connection is bad
/opt/couchbase/var/lib/couchbase/logs/goxdcr.log.2.gz:2021-09-29T05:43:12.992-07:00 ERRO GOXDCR.XmemNozzle: xmem_8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4_172.23.121.86:11210_0 batchGetMeta received fatal error and had to abort. Expected 500 responses, got 0 responses. err=Connection is bad

Logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632922287/collectinfo-2021-09-29T133129-ns_1%40172.23.99.25.zip