Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40832

[System Test] : 3 Rebalance failures due to error "CBAS0001: Datasets in different partitions have different DCP states"

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-2792
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Day : 1
      Iteration : 1

      In the first iteration of the test, after the analytics datasets were created at 2020-08-07T15:43, connect link Local command was issued at 2020-08-07T16:04.

      The rebalance operations related to analytics nodes were done at the following times -
      2020-08-07T16:28 (to add 172.23.123.33 as an analytics node to the cluster)
      2020-08-07T16:32 (to remove 172.23.120.75 analytics node from the cluster)
      2020-08-07T16:36 (to swap rebalance 172.23.106.134 with 172.23.120.75)

      All these 3 rebalance operations failed with the following errors - 

      4:29:45 PM   7 Aug, 2020
      		 
      Rebalance exited with reason {service_rebalance_failed,cbas,
      		 
      {worker_died,
      		 
      {'EXIT',<0.5164.43>,
      		 
      {rebalance_failed,
      		 
      {service_error,
      		 
      <<"Rebalance e1b09237a677feac89ab65e39b334f21 failed: CBAS0001: Datasets in different partitions have different DCP states. Mutations needed to catch up = 2659. User action: Try again later">>}}}}}.
      		 
      Rebalance Operation Id = 071fc9c69eaf6e37b9da4317bd499d3f


      4:32:59 PM   7 Aug, 2020
      		 
      Rebalance exited with reason {service_rebalance_failed,cbas,
      		 
      {worker_died,
      		 
      {'EXIT',<0.6394.45>,
      		 
      {rebalance_failed,
      		 
      {service_error,
      		 
      <<"Rebalance 6d5b750d9062033a8c584e35b6520cce failed: CBAS0001: Datasets in different partitions have different DCP states. Mutations needed to catch up = 3492. User action: Try again later">>}}}}}.
      		 
      Rebalance Operation Id = 94429177eeb3267a47b62f0452bbc1e6


      4:37:13 PM   7 Aug, 2020
      		 
      Rebalance exited with reason {service_rebalance_failed,cbas,
      		 
      {worker_died,
      		 
      {'EXIT',<0.30822.47>,
      		 
      {rebalance_failed,
      		 
      {service_error,
      		 
      <<"Rebalance 5642abd6ca5e13ebed78700e8deb506f failed: CBAS0001: Datasets in different partitions have different DCP states. Mutations needed to catch up = 3477. User action: Try again later">>}}}}}.
      		 
      Rebalance Operation Id = d59ccdc564924c83add7dae63d958ecd

      Nodes with cbas service : 172.23.106.134, 172.23.120.74, 172.23.120.75, 172.23.120.81, 172.23.123.33
      Master node : 172.23.97.74

      Attachments

        Issue Links

          links to

          Web Link Fix commit

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty