Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47169

[high-bucket] - 30 multi bucket test rebalance failed with buckets_cleanup_failed error

    XMLWordPrintable

Details

    Description

      Environment: 7.0.0-5295
      Test : 30 bucket test with all the components 
      Failed at : Rebalance step 
      Error message: 

      completionMessage":"Rebalance exited with reason
      Unknown macro:
      Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
      ."}

      Link to the job : http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/ 

      Steps of the test :

      1. Load the buckets with documents 
      2. Create n1ql indexes 
      3. Initialise XDCR (init_only_xdcr() )
      4. Creating the eventing functions
      5. Creating FTS indexes 
      6. Creating Analytics dataset
      7. Running rebalance for each phase as follows :
        1. KV rebalance 
          1. Rebalance in with mutations
          2. Rebalance swap 
          3. Rebalance out 
        2. Index rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        3. Eventing rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        4. CBAS rebalance 
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
      8. Backup
      9. FTS swap rebalance

      The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
      Cluster setup and the cluster details are mentioned in the screenshot attached below.

      Attachments

        1. image-2021-08-19-18-50-23-794.png
          image-2021-08-19-18-50-23-794.png
          295 kB
        2. image-2021-08-19-19-36-41-983.png
          image-2021-08-19-19-36-41-983.png
          682 kB
        3. loadavg.png
          loadavg.png
          148 kB
        4. screenshot-1.png
          screenshot-1.png
          42 kB
        5. screenshot-10.png
          screenshot-10.png
          32 kB
        6. screenshot-11.png
          screenshot-11.png
          47 kB
        7. screenshot-12.png
          screenshot-12.png
          44 kB
        8. screenshot-13.png
          screenshot-13.png
          38 kB
        9. screenshot-14.png
          screenshot-14.png
          38 kB
        10. screenshot-15.png
          screenshot-15.png
          45 kB
        11. screenshot-16.png
          screenshot-16.png
          37 kB
        12. screenshot-17.png
          screenshot-17.png
          37 kB
        13. screenshot-18.png
          screenshot-18.png
          38 kB
        14. screenshot-19.png
          screenshot-19.png
          33 kB
        15. screenshot-2.png
          screenshot-2.png
          35 kB
        16. screenshot-20.png
          screenshot-20.png
          39 kB
        17. Screenshot 2021-07-01 at 5.30.16 PM.png
          Screenshot 2021-07-01 at 5.30.16 PM.png
          75 kB
        18. Screenshot 2021-07-01 at 5.31.24 PM.png
          Screenshot 2021-07-01 at 5.31.24 PM.png
          33 kB
        19. Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          92 kB
        20. screenshot-3.png
          screenshot-3.png
          49 kB
        21. screenshot-4.png
          screenshot-4.png
          77 kB
        22. screenshot-5.png
          screenshot-5.png
          39 kB
        23. screenshot-6.png
          screenshot-6.png
          88 kB
        24. screenshot-7.png
          screenshot-7.png
          67 kB
        25. screenshot-8.png
          screenshot-8.png
          31 kB
        26. screenshot-9.png
          screenshot-9.png
          29 kB
        27. slow_fsyncs_1s_rate.png
          slow_fsyncs_1s_rate.png
          150 kB
        28. slow_fsyncs_1s.png
          slow_fsyncs_1s.png
          124 kB
        29. slow_fsyncs_5s.png
          slow_fsyncs_5s.png
          114 kB

        Issue Links

          For Gerrit Dashboard: MB-47169
          # Subject Branch Project Status CR V

          Activity

            jyotsna.nayak Jyotsna Nayak added a comment - - edited

            Murtadha Hubail  , I have run the test as mentioned in the comment above ; with the parameter set to 2173600 Bytes.
            The test is failing at after rebalancing all the components ; with the following error
            The cluster is not balanced
            Upon checking the rebalance logs , this is the message printed 

            {"stageInfo":{"analytics":{"totalProgress":2.484999999999952e-11,"perNodeProgress":

            {"ns_1@172.23.99.160":2.484999999999952e-13,"ns_1@172.23.96.23":2.484999999999952e-13}

            ,"startTime":"2022-03-30T18:04:08.826-07:00","completedTime":false,"timeTaken":2554572},"eventing":{"startTime":false,"completedTime":false,"timeTaken":false},"search":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.96.20":1}

            ,"startTime":"2022-03-30T18:04:05.482-07:00","completedTime":"2022-03-30T18:04:05.936-07:00","timeTaken":453},"index":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.96.15":1,"ns_1@172.23.96.19":1}

            ,"startTime":"2022-03-30T18:04:05.936-07:00","completedTime":"2022-03-30T18:04:08.826-07:00","timeTaken":2890},"data":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.99.157":1,"ns_1@172.23.99.158":1,"ns_1@172.23.99.159":1}

            ,"startTime":"2022-03-30T18:03:55.918-07:00","completedTime":"2022-03-30T18:04:05.482-07:00","timeTaken":9565},"query":{"startTime":false,"completedTime":false,"timeTaken":false}},"rebalanceId":"9d7d027beca1eaf5d1746604e115a43f","nodesInfo":

            {"active_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"keep_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]}

            ,"masterNode":"ns_1@172.23.99.157","startTime":"2022-03-30T18:03:55.913-07:00","completedTime":"2022-03-30T18:46:43.398-07:00","timeTaken":2567486,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {worker_died,\n                               {'EXIT',<0.25460.435>,\n                                {rebalance_failed,\n                                

            {service_error,\n                                  <<\"Rebalance 5692dee195b5f22cd3fb646ea3a742a8 failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1738. User action: Try again later\">>}

            }}}}."}

            Link to the job :  http://perf.jenkins.couchbase.com/job/themis_multibucket/121/

            logs:

            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.15.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.19.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.20.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.23.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.97.177.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.157.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.158.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.159.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.160.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.161.zip
            https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/tools.zip

            jyotsna.nayak Jyotsna Nayak added a comment - - edited Murtadha Hubail   , I have run the test as mentioned in the comment above ; with the parameter set to 2173600 Bytes. The test is failing at after rebalancing all the components ; with the following error The cluster is not balanced Upon checking the rebalance logs , this is the message printed  {"stageInfo":{"analytics":{"totalProgress":2.484999999999952e-11,"perNodeProgress": {"ns_1@172.23.99.160":2.484999999999952e-13,"ns_1@172.23.96.23":2.484999999999952e-13} ,"startTime":"2022-03-30T18:04:08.826-07:00","completedTime":false,"timeTaken":2554572},"eventing":{"startTime":false,"completedTime":false,"timeTaken":false},"search":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.96.20":1} ,"startTime":"2022-03-30T18:04:05.482-07:00","completedTime":"2022-03-30T18:04:05.936-07:00","timeTaken":453},"index":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.96.15":1,"ns_1@172.23.96.19":1} ,"startTime":"2022-03-30T18:04:05.936-07:00","completedTime":"2022-03-30T18:04:08.826-07:00","timeTaken":2890},"data":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.99.157":1,"ns_1@172.23.99.158":1,"ns_1@172.23.99.159":1} ,"startTime":"2022-03-30T18:03:55.918-07:00","completedTime":"2022-03-30T18:04:05.482-07:00","timeTaken":9565},"query":{"startTime":false,"completedTime":false,"timeTaken":false}},"rebalanceId":"9d7d027beca1eaf5d1746604e115a43f","nodesInfo": {"active_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"keep_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]} ,"masterNode":"ns_1@172.23.99.157","startTime":"2022-03-30T18:03:55.913-07:00","completedTime":"2022-03-30T18:46:43.398-07:00","timeTaken":2567486,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {worker_died,\n                               {'EXIT',<0.25460.435>,\n                                {rebalance_failed,\n                                 {service_error,\n                                  <<\"Rebalance 5692dee195b5f22cd3fb646ea3a742a8 failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1738. User action: Try again later\">>} }}}}."} Link to the job :  http://perf.jenkins.couchbase.com/job/themis_multibucket/121/ logs: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.15.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.19.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.20.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.96.23.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.97.177.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.157.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.158.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.159.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.160.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/172.23.99.161.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis_multibucket-121/tools.zip

            Jyotsna Nayak,

            This is expected when the DCP stream is disconnected from Analytics ungracefully (e.g. as a result of a KV topology change). As the rebalance failure message suggests, some data partitions are 1738 mutations behind. It usually takes less than a minute for all partitions to catch up to the same DCP state. If you try the rebalance after a minute or so, the rebalance should proceed.

            murtadha.hubail Murtadha Hubail added a comment - Jyotsna Nayak , This is expected when the DCP stream is disconnected from Analytics ungracefully (e.g. as a result of a KV topology change). As the rebalance failure message suggests, some data partitions are 1738 mutations behind. It usually takes less than a minute for all partitions to catch up to the same DCP state. If you try the rebalance after a minute or so, the rebalance should proceed.

            This test has run from end to end ; and the cluster seemed to be balanced on the UI front a few mins after the test completed the run . 
            Link to the job :  http://perf.jenkins.couchbase.com/job/themis_multibucket/121/
            Will have a rerun of the test after increasing the sleep time  in between the rebalances is increased.
            The issue due to which this bug was initially filed is no longer observed. 

            jyotsna.nayak Jyotsna Nayak added a comment - This test has run from end to end ; and the cluster seemed to be balanced on the UI front a few mins after the test completed the run .  Link to the job :  http://perf.jenkins.couchbase.com/job/themis_multibucket/121/ Will have a rerun of the test after increasing the sleep time  in between the rebalances is increased. The issue due to which this bug was initially filed is no longer observed. 
            wayne Wayne Siu added a comment -

            Jyotsna NayakMurtadha Hubail
            I'm closing this ticket based on latest updates. (origin issue reported is no longer observed).
            Please open a new ticket should there is a new issue from the re-run with a new sleep time. Thanks.

            wayne Wayne Siu added a comment - Jyotsna Nayak Murtadha Hubail I'm closing this ticket based on latest updates. (origin issue reported is no longer observed). Please open a new ticket should there is a new issue from the re-run with a new sleep time. Thanks.
            jyotsna.nayak Jyotsna Nayak added a comment - - edited

            Analysis after increasing the amount of sleep :

            I have rerun the test after increasing the sleep between the rebalances from 1 hour to 
            1.  2 hours (test failed due to 6 cbas mutations left to catch up ; link to job: here 
            Error message: 

            {"stageInfo":{"analytics":{"totalProgress":5.700000000000002e-13,"perNodeProgress":

            {"ns_1@172.23.99.160":5.700000000000002e-15,"ns_1@172.23.96.23":5.700000000000002e-15}

            ,"startTime":"2022-04-12T20:26:36.001-07:00","completedTime":false,"timeTaken":56506},"eventing":{"startTime":false,"completedTime":false,"timeTaken":false},"search":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.96.20":1}

            ,"startTime":"2022-04-12T20:26:32.355-07:00","completedTime":"2022-04-12T20:26:32.869-07:00","timeTaken":514},"index":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.96.15":1,"ns_1@172.23.96.19":1}

            ,"startTime":"2022-04-12T20:26:32.869-07:00","completedTime":"2022-04-12T20:26:36.001-07:00","timeTaken":3132},"data":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.99.157":1,"ns_1@172.23.99.158":1,"ns_1@172.23.99.159":1}

            ,"startTime":"2022-04-12T20:26:22.889-07:00","completedTime":"2022-04-12T20:26:32.355-07:00","timeTaken":9466},"query":{"startTime":false,"completedTime":false,"timeTaken":false}},"rebalanceId":"fef9a523cd142ca550b5671cb67f02ec","nodesInfo":

            {"active_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"keep_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]}

            ,"masterNode":"ns_1@172.23.99.157","startTime":"2022-04-12T20:26:22.880-07:00","completedTime":"2022-04-12T20:27:32.508-07:00","timeTaken":69628,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {worker_died,\n                               {'EXIT',<0.23164.614>,\n                                {rebalance_failed,\n                                 {service_error,\n                                  <<\"Rebalance cf90e012469a96b7555ad9eb9a0902cc failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 6. User action: Try again later\">>}}}}}."}

            2. 3 hours (test failed due to 1 cbas mutations left to catch up ; link to the job : here )
            Error message:
            {"stageInfo":{"analytics":{"totalProgress":5.729979539608404,"perNodeProgress":

            {"ns_1@172.23.99.160":0.05729979539608404,"ns_1@172.23.96.23":0.05729979539608404}

            ,"startTime":"2022-04-21T22:45:58.519-07:00","completedTime":false,"timeTaken":481388},"eventing":{"startTime":false,"completedTime":false,"timeTaken":false},"search":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.96.20":1}

            ,"startTime":"2022-04-21T22:45:54.745-07:00","completedTime":"2022-04-21T22:45:55.266-07:00","timeTaken":520},"index":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.96.15":1,"ns_1@172.23.96.19":1}

            ,"startTime":"2022-04-21T22:45:55.266-07:00","completedTime":"2022-04-21T22:45:58.519-07:00","timeTaken":3253},"data":{"totalProgress":100,"perNodeProgress":

            {"ns_1@172.23.99.157":1,"ns_1@172.23.99.158":1,"ns_1@172.23.99.159":1}

            ,"startTime":"2022-04-21T22:45:45.579-07:00","completedTime":"2022-04-21T22:45:54.745-07:00","timeTaken":9166},"query":{"startTime":false,"completedTime":false,"timeTaken":false}},"rebalanceId":"a484886399b811651e3c3a8386bdb95c","nodesInfo":

            {"active_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"keep_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]}

            ,"masterNode":"ns_1@172.23.99.157","startTime":"2022-04-21T22:45:45.574-07:00","completedTime":"2022-04-21T22:53:59.906-07:00","timeTaken":494332,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {worker_died,\n                               {'EXIT',<0.17599.784>,\n                                {rebalance_failed,\n                                 {service_error,\n                                  <<\"Rebalance 861ea35e761c76836acfa59ee14411da failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1. User action: Try again later\">>}}}}}."} 

            jyotsna.nayak Jyotsna Nayak added a comment - - edited Analysis after increasing the amount of sleep : I have rerun the test after increasing the sleep between the rebalances from 1 hour to  1.   2 hours (test failed due to 6 cbas mutations left to catch up ; link to job:  here   Error message:   {"stageInfo":{"analytics":{"totalProgress":5.700000000000002e-13,"perNodeProgress": {"ns_1@172.23.99.160":5.700000000000002e-15,"ns_1@172.23.96.23":5.700000000000002e-15} ,"startTime":"2022-04-12T20:26:36.001-07:00","completedTime":false,"timeTaken":56506},"eventing":{"startTime":false,"completedTime":false,"timeTaken":false},"search":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.96.20":1} ,"startTime":"2022-04-12T20:26:32.355-07:00","completedTime":"2022-04-12T20:26:32.869-07:00","timeTaken":514},"index":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.96.15":1,"ns_1@172.23.96.19":1} ,"startTime":"2022-04-12T20:26:32.869-07:00","completedTime":"2022-04-12T20:26:36.001-07:00","timeTaken":3132},"data":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.99.157":1,"ns_1@172.23.99.158":1,"ns_1@172.23.99.159":1} ,"startTime":"2022-04-12T20:26:22.889-07:00","completedTime":"2022-04-12T20:26:32.355-07:00","timeTaken":9466},"query":{"startTime":false,"completedTime":false,"timeTaken":false}},"rebalanceId":"fef9a523cd142ca550b5671cb67f02ec","nodesInfo": {"active_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"keep_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]} ,"masterNode":"ns_1@172.23.99.157","startTime":"2022-04-12T20:26:22.880-07:00","completedTime":"2022-04-12T20:27:32.508-07:00","timeTaken":69628,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {worker_died,\n                               {'EXIT',<0.23164.614>,\n                                {rebalance_failed,\n                                 {service_error,\n                                  <<\"Rebalance cf90e012469a96b7555ad9eb9a0902cc failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 6 . User action: Try again later\">>}}}}}."} 2. 3 hours (test failed due to 1 cbas mutations left to catch up ; link to the job : here  ) Error message: {"stageInfo":{"analytics":{"totalProgress":5.729979539608404,"perNodeProgress": {"ns_1@172.23.99.160":0.05729979539608404,"ns_1@172.23.96.23":0.05729979539608404} ,"startTime":"2022-04-21T22:45:58.519-07:00","completedTime":false,"timeTaken":481388},"eventing":{"startTime":false,"completedTime":false,"timeTaken":false},"search":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.96.20":1} ,"startTime":"2022-04-21T22:45:54.745-07:00","completedTime":"2022-04-21T22:45:55.266-07:00","timeTaken":520},"index":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.96.15":1,"ns_1@172.23.96.19":1} ,"startTime":"2022-04-21T22:45:55.266-07:00","completedTime":"2022-04-21T22:45:58.519-07:00","timeTaken":3253},"data":{"totalProgress":100,"perNodeProgress": {"ns_1@172.23.99.157":1,"ns_1@172.23.99.158":1,"ns_1@172.23.99.159":1} ,"startTime":"2022-04-21T22:45:45.579-07:00","completedTime":"2022-04-21T22:45:54.745-07:00","timeTaken":9166},"query":{"startTime":false,"completedTime":false,"timeTaken":false}},"rebalanceId":"a484886399b811651e3c3a8386bdb95c","nodesInfo": {"active_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"keep_nodes":["ns_1@172.23.99.157","ns_1@172.23.99.158","ns_1@172.23.99.159","ns_1@172.23.96.19","ns_1@172.23.96.15","ns_1@172.23.97.177","ns_1@172.23.96.23","ns_1@172.23.96.20","ns_1@172.23.99.160"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]} ,"masterNode":"ns_1@172.23.99.157","startTime":"2022-04-21T22:45:45.574-07:00","completedTime":"2022-04-21T22:53:59.906-07:00","timeTaken":494332,"completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {worker_died,\n                               {'EXIT',<0.17599.784>,\n                                {rebalance_failed,\n                                 {service_error,\n                                  <<\"Rebalance 861ea35e761c76836acfa59ee14411da failed: CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1 . User action: Try again later\">>}}}}}."} 

            People

              jyotsna.nayak Jyotsna Nayak
              jyotsna.nayak Jyotsna Nayak
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty