[System Test] Analytics rebalance failure in longevity - CBAS0001: Datasets in different partitions have different DCP states

7.0.0-4587

Test:
-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
1st iteration

From .104.232:

Rebalance exited with reason {service_rebalance_failed,cbas,

{worker_died,

{'EXIT',<0.11409.267>,

{rebalance_failed,

{service_error,

<<"Rebalance f35b1607d1301f5b642b75263e0ab3bd failed: CBAS0001: Datasets in different partitions have different DCP states. Mutations needed to catch up = 220. User action: Try again later">>}}}}}.

Rebalance Operation Id = 43d5ad92ce042525ca1377aab62e7349

Logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.104.15.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.104.165.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.104.214.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.104.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.104.244.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.104.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.102.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.118.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.164.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.183.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.206.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.210.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.29.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.61.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.62.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.90.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.105.93.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.191.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.207.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.209.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.225.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614808245/collectinfo-2021-03-03T215047-ns_1%40172.23.106.70.zip