Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46930

[BP 6.6.3][System Test][Analytics] Rebalance failures when node is removed - CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1033950. User action: Try again later

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • CX Sprint 251

    Description

      7.0.0-5219

      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3
      Iteration 1

      Two rebalance failures observed when .120.75 is removed

      2021-05-22T03:40:09.357-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.74) - Rebalance exited with reason {service_rebalance_failed,cbas,
      		 
                                    {worker_died,
      		 
                                     {'EXIT',<0.26646.220>,
      		 
                                      {rebalance_failed,
      		 
                                       {service_error,
      		 
                                        <<"Rebalance 2ea50a80af7a818a3b251f3e3a8ce61a failed: HYR0003: Failure on node 172.23.120.74:8091 (14f42078585c84fad8336bb31266666b)">>}}}}}.
      		 
      Rebalance Operation Id = 68c4cca8e35f2e7fd85410abfc5dbc0b


      2021-05-22T03:47:53.148-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.74) - Rebalance exited with reason {service_rebalance_failed,cbas,
      		 
                                    {worker_died,
      		 
                                     {'EXIT',<0.27397.228>,
      		 
                                      {rebalance_failed,
      		 
                                       {service_error,
      		 
                                        <<"Rebalance 0d27d5ac35b9e8f54b05bb440ddb809c failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1033950. User action: Try again later">>}}}}}.
      		 
      Rebalance Operation Id = f8238a9af9b31b60765579a13e0173a3

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.106.136.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.106.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.149.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.151.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621685083/collectinfo-2021-05-22T120444-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-46465 [System Test][Analytics] Rebalance failures when node is removed - CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1033950. User action: Try again later

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty