Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49512

[Magma] - Cleaning up of the cluster fails with "Rebalance exited with reason {buckets_shutdown_wait_failed"

    XMLWordPrintable

Details

    • Triaged
    • Centos 64-bit
    • 1
    • No
    • KV 2021-Dec, KV 2022-Feb, KV March-22

    Description

      Script to Repro

      This can happen in the tearDown part of the any test. So, in tearDown method we drop all the buckets and remove all the nodes in the cluster. This fails as shown below.

      172.23.120.206 10:05:03 PM 11 Nov, 2021 ( 2021-11-11T22:05:03.228-08:00 )

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      		 
      [{'ns_1@172.23.120.206',
      		 
      {'EXIT',
      		 
      {old_buckets_shutdown_wait_failed,
      		 
      ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
      		 
      Rebalance Operation Id = dbb8d76ebc02c654f2c23fbbabac68e9

      Even retried rebalance failed.
      172.23.120.206 10:06:28 PM 11 Nov, 2021

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      		 
      [{'ns_1@172.23.120.206',
      		 
      {'EXIT',
      		 
      {old_buckets_shutdown_wait_failed,
      		 
      ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
      		 
      Rebalance Operation Id = 1754a1e783a53551c9f546338cebb3d7

      Based on the failures it does look like the previously dropped bucket too longer than expected to get deleted.

      172.23.104.186 10:03:32 PM 11 Nov, 2021

      Shutting down bucket "-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000" on 'ns_1@172.23.104.186' for deletion

      Maybe we need to figure out a way to disable the rebalance button until the bucket is fully deleted.

      cbcollect_info attached.

      Attachments

        1. 172.23.100.38.zip
          24.32 MB
        2. 172.23.100.39.zip
          28.18 MB
        3. consoleText_MB-49512_rerun.txt
          320 kB
        4. consoleText_MB-49512_run2_2211.txt
          3.08 MB
        5. screenshot-1.png
          screenshot-1.png
          36 kB
        6. Screenshot 2022-02-26 at 4.19.57 PM.png
          Screenshot 2022-02-26 at 4.19.57 PM.png
          294 kB
        7. UI_MB-49512.png
          UI_MB-49512.png
          595 kB

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-51132 SET Latency (99.9th percentile or higher) during Rebalance-swap increased on build 2335

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-48872 Bucket deletion can be blocked if another Bucket is performing higher-priority long-running tasks (e.g. Rollback)

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-51477 [Magma] Rebalance exited with reason buckets_shutdown_wait_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-50988 Rescheduled Compaction tasks do not obey the concurrency limit

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apaar.gupta Apaar Gupta
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There is 1 open Gerrit change

                  PagerDuty