Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49512

[Magma] - Cleaning up of the cluster fails with "Rebalance exited with reason {buckets_shutdown_wait_failed"

    XMLWordPrintable

Details

    • Triaged
    • Centos 64-bit
    • 1
    • No
    • KV 2021-Dec, KV 2022-Feb, KV March-22

    Description

      Script to Repro

      This can happen in the tearDown part of the any test. So, in tearDown method we drop all the buckets and remove all the nodes in the cluster. This fails as shown below.
      

      172.23.120.206 10:05:03 PM 11 Nov, 2021 ( 2021-11-11T22:05:03.228-08:00 )

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      [{'ns_1@172.23.120.206',
      {'EXIT',
      {old_buckets_shutdown_wait_failed,
      ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
      Rebalance Operation Id = dbb8d76ebc02c654f2c23fbbabac68e9
      

      Even retried rebalance failed.
      172.23.120.206 10:06:28 PM 11 Nov, 2021

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      [{'ns_1@172.23.120.206',
      {'EXIT',
      {old_buckets_shutdown_wait_failed,
      ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
      Rebalance Operation Id = 1754a1e783a53551c9f546338cebb3d7
      

      Based on the failures it does look like the previously dropped bucket too longer than expected to get deleted.

      172.23.104.186 10:03:32 PM 11 Nov, 2021

      Shutting down bucket "-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000" on 'ns_1@172.23.104.186' for deletion
      

      Maybe we need to figure out a way to disable the rebalance button until the bucket is fully deleted.

      cbcollect_info attached.

      Attachments

        1. 172.23.100.38.zip
          24.32 MB
        2. 172.23.100.39.zip
          28.18 MB
        3. consoleText_MB-49512_rerun.txt
          320 kB
        4. consoleText_MB-49512_run2_2211.txt
          3.08 MB
        5. screenshot-1.png
          screenshot-1.png
          36 kB
        6. Screenshot 2022-02-26 at 4.19.57 PM.png
          Screenshot 2022-02-26 at 4.19.57 PM.png
          294 kB
        7. UI_MB-49512.png
          UI_MB-49512.png
          595 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Balakumaran.Gopal Balakumaran Gopal created issue -
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Field Original Value New Value
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] James Harrison [ james.harrison ]
            james.harrison James Harrison made changes -
            Sprint KV 2021-Dec [ 1906 ]
            james.harrison James Harrison made changes -
            Description +Script to Repro+
            {noformat}
            This can happen in the tearDown part of the any test. So, in tearDown method we drop all the buckets and remove all the nodes in the cluster. This fails as shown below.
            {noformat}

            +172.23.120.206 10:05:03 PM 11 Nov, 2021+
            {noformat}
            Rebalance exited with reason {buckets_shutdown_wait_failed,
            [{'ns_1@172.23.120.206',
            {'EXIT',
            {old_buckets_shutdown_wait_failed,
            ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
            Rebalance Operation Id = dbb8d76ebc02c654f2c23fbbabac68e9
            {noformat}

            Even retried rebalance failed.
            +172.23.120.206 10:06:28 PM 11 Nov, 2021+
            {noformat}
            Rebalance exited with reason {buckets_shutdown_wait_failed,
            [{'ns_1@172.23.120.206',
            {'EXIT',
            {old_buckets_shutdown_wait_failed,
            ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
            Rebalance Operation Id = 1754a1e783a53551c9f546338cebb3d7
            {noformat}

            Based on the failures it does look like the previously dropped bucket too longer than expected to get deleted.

            +172.23.104.186 10:03:32 PM 11 Nov, 2021+
            {noformat}
            Shutting down bucket "-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000" on 'ns_1@172.23.104.186' for deletion
            {noformat}

            Maybe we need to figure out a way to disable the rebalance button until the bucket is fully deleted.

            cbcollect_info attached.
            +Script to Repro+
            {noformat}
            This can happen in the tearDown part of the any test. So, in tearDown method we drop all the buckets and remove all the nodes in the cluster. This fails as shown below.
            {noformat}

            +172.23.120.206 10:05:03 PM 11 Nov, 2021+ ( 2021-11-11T22:05:03.228-08:00 )
            {noformat}
            Rebalance exited with reason {buckets_shutdown_wait_failed,
            [{'ns_1@172.23.120.206',
            {'EXIT',
            {old_buckets_shutdown_wait_failed,
            ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
            Rebalance Operation Id = dbb8d76ebc02c654f2c23fbbabac68e9
            {noformat}

            Even retried rebalance failed.
            +172.23.120.206 10:06:28 PM 11 Nov, 2021+
            {noformat}
            Rebalance exited with reason {buckets_shutdown_wait_failed,
            [{'ns_1@172.23.120.206',
            {'EXIT',
            {old_buckets_shutdown_wait_failed,
            ["-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000"]}}}]}.
            Rebalance Operation Id = 1754a1e783a53551c9f546338cebb3d7
            {noformat}

            Based on the failures it does look like the previously dropped bucket too longer than expected to get deleted.

            +172.23.104.186 10:03:32 PM 11 Nov, 2021+
            {noformat}
            Shutting down bucket "-6AT-Evkts1eHVShDkwV6uJIF5j5BxpFu2DwiLTw0PnB0bYy-33-378000" on 'ns_1@172.23.104.186' for deletion
            {noformat}

            Maybe we need to figure out a way to disable the rebalance button until the bucket is fully deleted.

            cbcollect_info attached.
            owend Daniel Owen made changes -
            Sprint KV 2021-Dec [ 1906 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Assignee James Harrison [ james.harrison ] Daniel Owen [ owend ]
            richard.demellow Richard deMellow made changes -
            Assignee Daniel Owen [ owend ] Richard deMellow [ richard.demellow ]
            owend Daniel Owen made changes -
            Sprint KV 2021-Dec [ 1906 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            richard.demellow Richard deMellow made changes -
            Attachment screenshot-1.png [ 171995 ]
            richard.demellow Richard deMellow made changes -
            Component/s couchbase-bucket [ 10173 ]
            Component/s storage-engine [ 10175 ]
            richard.demellow Richard deMellow made changes -
            Assignee Richard deMellow [ richard.demellow ] John Liang [ jliang ]
            richard.demellow Richard deMellow made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            richard.demellow Richard deMellow made changes -
            Assignee John Liang [ jliang ] Sarath Lakshman [ sarath ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Assignee Sarath Lakshman [ sarath ] Srinath Duvuru [ srinath.duvuru ]
            srinath.duvuru Srinath Duvuru made changes -
            Assignee Srinath Duvuru [ srinath.duvuru ] Apaar Gupta [ apaar.gupta ]
            srinath.duvuru Srinath Duvuru made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            apaar.gupta Apaar Gupta made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Attachment consoleText_MB-49512_run2_2211.txt [ 176391 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            apaar.gupta Apaar Gupta made changes -
            Assignee Apaar Gupta [ apaar.gupta ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Component/s couchbase-bucket [ 10173 ]
            owend Daniel Owen made changes -
            Rank Ranked lower
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ben Huddleston [ ben.huddleston ]
            owend Daniel Owen made changes -
            Sprint KV 2021-Dec [ 1906 ] KV 2021-Dec, KV 2022-Feb [ 1906, 2002 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Component/s storage-engine [ 10175 ]
            owend Daniel Owen made changes -
            Due Date 24/Feb/22
            drigby Dave Rigby made changes -
            Link This issue relates to MB-50988 [ MB-50988 ]
            ben.huddleston Ben Huddleston made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            ben.huddleston Ben Huddleston made changes -
            Link This issue is duplicated by MB-48872 [ MB-48872 ]
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Balakumaran Gopal [ balakumaran.gopal ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Daniel Owen [ owend ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Attachment UI_MB-49512.png [ 178114 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ben Huddleston [ ben.huddleston ]
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Dave Finlay [ dfinlay ]
            steve.watanabe Steve Watanabe made changes -
            Assignee Dave Finlay [ dfinlay ] Steve Watanabe [ steve.watanabe ]
            steve.watanabe Steve Watanabe made changes -
            Assignee Steve Watanabe [ steve.watanabe ] Ben Huddleston [ ben.huddleston ]
            steve.watanabe Steve Watanabe made changes -
            Environment Enterprise Edition 7.1.0 build 1694 ‧ Enterprise Edition 7.1.0 build 1694 ‧
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Balakumaran Gopal [ balakumaran.gopal ]
            owend Daniel Owen made changes -
            Sprint KV 2021-Dec, KV 2022-Feb [ 1906, 2002 ] KV 2021-Dec [ 1906 ]
            owend Daniel Owen made changes -
            Rank Ranked lower
            ben.huddleston Ben Huddleston made changes -
            Due Date 24/Feb/22 25/Feb/22
            ben.huddleston Ben Huddleston made changes -
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Ben Huddleston [ ben.huddleston ]
            drigby Dave Rigby made changes -
            Sprint KV 2021-Dec [ 1906 ] KV 2021-Dec, KV 2022-Feb [ 1906, 2002 ]
            drigby Dave Rigby made changes -
            Rank Ranked higher
            ben.huddleston Ben Huddleston made changes -
            Labels functional-test magma functional-test magma releasenote
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Attachment consoleText_MB-49512_rerun.txt [ 178436 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            ben.huddleston Ben Huddleston made changes -
            Due Date 25/Feb/22 01/Mar/22
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Balakumaran Gopal [ balakumaran.gopal ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Ben Huddleston [ ben.huddleston ]
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Balakumaran Gopal [ balakumaran.gopal ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Ben Huddleston [ ben.huddleston ]
            ben.huddleston Ben Huddleston made changes -
            Due Date 01/Mar/22 02/Mar/22
            ben.huddleston Ben Huddleston made changes -
            Due Date 02/Mar/22 04/Mar/22
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Balakumaran Gopal [ balakumaran.gopal ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            wayne Wayne Siu made changes -
            Link This issue causes MB-51132 [ MB-51132 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Ben Huddleston [ ben.huddleston ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            ben.huddleston Ben Huddleston made changes -
            Due Date 04/Mar/22 09/Mar/22
            owend Daniel Owen made changes -
            Sprint KV 2021-Dec, KV 2022-Feb [ 1906, 2002 ] KV 2021-Dec, KV 2022-Feb, KV Post-Neo 2 [ 1906, 2002, 2050 ]
            owend Daniel Owen made changes -
            Due Date 09/Mar/22 10/Mar/22
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Balakumaran Gopal [ balakumaran.gopal ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            pavithra.mahamani Pavithra Mahamani (Inactive) made changes -
            Attachment 172.23.100.38.zip [ 179628 ]
            pavithra.mahamani Pavithra Mahamani (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            pavithra.mahamani Pavithra Mahamani (Inactive) made changes -
            Assignee Balakumaran Gopal [ balakumaran.gopal ] Ben Huddleston [ ben.huddleston ]
            ritam.sharma Ritam Sharma made changes -
            Labels functional-test magma releasenote affects-neo-testing functional-test magma releasenote
            pavithra.mahamani Pavithra Mahamani (Inactive) made changes -
            Attachment 172.23.100.39.zip [ 179629 ]
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Sarath Lakshman [ sarath ]
            ben.huddleston Ben Huddleston made changes -
            Component/s couchbase-bucket [ 10173 ]
            Component/s storage-engine [ 10175 ]
            sarath Sarath Lakshman made changes -
            Assignee Sarath Lakshman [ sarath ] Apaar Gupta [ apaar.gupta ]
            pavithra.mahamani Pavithra Mahamani (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Closed [ 6 ]
            pavithra.mahamani Pavithra Mahamani (Inactive) made changes -
            Link This issue relates to MB-51477 [ MB-51477 ]

            People

              apaar.gupta Apaar Gupta
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There is 1 open Gerrit change

                  PagerDuty