Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49512

[Magma] - Cleaning up of the cluster fails with "Rebalance exited with reason {buckets_shutdown_wait_failed"

    XMLWordPrintable

Details

    • Triaged
    • Centos 64-bit
    • 1
    • No
    • KV 2021-Dec, KV 2022-Feb, KV March-22

    Description

      Script to Repro

      This can happen in the tearDown part of the any test. So, in tearDown method we drop all the buckets and remove all the nodes in the cluster. This fails as shown below.
      

      172.23.120.206 10:05:03 PM 11 Nov, 2021 ( 2021-11-11T22:05:03.228-08:00 )

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      [{'ns_1@172.23.120.206',
      {'EXIT',
      {old_buckets_shutdown_wait_failed,
      ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
      Rebalance Operation Id = dbb8d76ebc02c654f2c23fbbabac68e9
      

      Even retried rebalance failed.
      172.23.120.206 10:06:28 PM 11 Nov, 2021

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      [{'ns_1@172.23.120.206',
      {'EXIT',
      {old_buckets_shutdown_wait_failed,
      ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
      Rebalance Operation Id = 1754a1e783a53551c9f546338cebb3d7
      

      Based on the failures it does look like the previously dropped bucket too longer than expected to get deleted.

      172.23.104.186 10:03:32 PM 11 Nov, 2021

      Shutting down bucket "-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000" on 'ns_1@172.23.104.186' for deletion
      

      Maybe we need to figure out a way to disable the rebalance button until the bucket is fully deleted.

      cbcollect_info attached.

      Attachments

        1. 172.23.100.38.zip
          24.32 MB
        2. 172.23.100.39.zip
          28.18 MB
        3. consoleText_MB-49512_rerun.txt
          320 kB
        4. consoleText_MB-49512_run2_2211.txt
          3.08 MB
        5. screenshot-1.png
          screenshot-1.png
          36 kB
        6. Screenshot 2022-02-26 at 4.19.57 PM.png
          Screenshot 2022-02-26 at 4.19.57 PM.png
          294 kB
        7. UI_MB-49512.png
          UI_MB-49512.png
          595 kB

        Issue Links

          For Gerrit Dashboard: MB-49512
          # Subject Branch Project Status CR V

          Activity

            No work has yet been logged on this issue.

            People

              apaar.gupta Apaar Gupta
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There is 1 open Gerrit change

                  PagerDuty