Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54664

Rebalance failed in Capella Analytics rebalance-out runs

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes
    • Analytics Sprint 8, Analytics Sprint 9, Analytics Sprint 10

    Description

      I did two rebalance out runs on Capella. I changed the number of analytics nodes from 4 to 3. I saw the error messages in both runs.

      "completionMessage":"Rebalance exited with reason {service_rebalance_failed,cbas, {worker_died, {'EXIT',<0.6.13>, {rebalance_failed,

      {service_error, <<\"Rebalance 50fd686873683e9de3f092518ce97f40 failed: see analytics_info.log for details\">>}

      }}}}."}

       

      Build: 7.1.3-3479

      Job:

      http://perf.jenkins.couchbase.com/job/CapellaTester/125/

      http://perf.jenkins.couchbase.com/job/CapellaTester/126/

      Logs:

      Job 125:

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/0kll47tyhqob1n1a.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/9lwm3ylfnsnrqfi.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/jin9zugrq7jm66qm.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/n93pbbw1t5df4yq.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/pm5oro1to-2ronr.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/vsw1owcygrxi05rq.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-125/xzonqmouuwcmwfnq.douwyvwhacuiyk.sandbox.nonprod-project-avengers.com.zip

       

      Job 126:

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/amorn0jggmhsvynv.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/awty1op2jxr-xr0i.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/fv1mahwwnaartqd7.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/lzw4aifmeh6-dd2.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/qaop7pirgbgfkrbq.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/z0awhcoaqiqx76yr.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-CapellaTester-126/zjaxv1fyqijrqdm.pjz8t8trjgixctc0.sandbox.nonprod-project-avengers.com.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.5.0-3518 contains cbas-core commit 76e2b63 with commit message:
            MB-54664: Exclude replicated partitions from rebalance required check

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.5.0-3518 contains cbas-core commit 76e2b63 with commit message: MB-54664 : Exclude replicated partitions from rebalance required check

            Build couchbase-server-8.0.0-1212 contains cbas-core commit 76e2b63 with commit message:
            MB-54664: Exclude replicated partitions from rebalance required check

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1212 contains cbas-core commit 76e2b63 with commit message: MB-54664 : Exclude replicated partitions from rebalance required check
            ming.ho Ming Ho added a comment -

            We can produce a special image.  aws AMI or Azure image?  looking at the commit history 7.2.0-5068 could do the trick.  Bo-Chun Wang Murtadha Hubail can you confirm?

            ming.ho Ming Ho added a comment - We can produce a special image.  aws AMI or Azure image?  looking at the commit history 7.2.0-5068 could do the trick.   Bo-Chun Wang Murtadha Hubail can you confirm?

            Ming Ho,
            Yes, build 7.2.0-5068 contains the commits we are interested in testing. Those commits were also back ported to 7.1.4-3556. I will let Bo-Chun Wang decide on the preferred build and image type.

            murtadha.hubail Murtadha Hubail added a comment - Ming Ho , Yes, build 7.2.0-5068 contains the commits we are interested in testing. Those commits were also back ported to 7.1.4-3556. I will let Bo-Chun Wang decide on the preferred build and image type.
            ming.ho Ming Ho added a comment -

            Per discussion on slack with Wayne and Vikas, couchbase-cloud-server-7.2.0-5068 was published to AWS.  Bo-Chun Wang 

            ming.ho Ming Ho added a comment - Per discussion on slack with Wayne and Vikas, couchbase-cloud-server-7.2.0-5068 was published to AWS.   Bo-Chun Wang  

            People

              bo-chun.wang Bo-Chun Wang
              bo-chun.wang Bo-Chun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty