[System Test][XDCR] ~27M mutations remaining - needs investigation

7.1.0-1962

Test:
-test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
Scale 3
Iteration 1

~27M mutations remaining in XDCR - yest it was around 26M items - there has been rebalances stuck for long time due to another issue (MB-50008) - not sure if it is due to that

Rebalances stuck:

[2021-12-22T01:42:13-08:00, sequoiatools/couchbase-cli:7.1:e8160a] rebalance -c 172.23.108.103:8091 -u Administrator -p password
[2021-12-23T02:48:51-08:00, sequoiatools/cmd:88ed55] 60

[2021-12-23T03:06:31-08:00, sequoiatools/couchbase-cli:7.1:381453] rebalance -c 172.23.108.103:8091 -u Administrator -p password

Logs collected before the rebalance that started at 2021-12-22T01:42:13-08:00:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.105.111.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.120.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640163560/collectinfo-2021-12-22T085923-ns_1%40172.23.99.25.zip

Logs collected after that rebalance:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.105.111.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.120.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1640256819/collectinfo-2021-12-23T105341-ns_1%40172.23.99.25.zip