Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6710

Deleting bucket on a cluster gives error "Some nodes are still deleting bucket"

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0-beta-2
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Environment:
      build 1751 on Centos 64

      Description

      Steps to reproduce:

      1. Create bucket on a single node
      2. Load 50K items
      3. Rebalance in 3 node
      4. Wait for rebalance to complete and then delete the bucket.

      Bucket deletion gives error(see screenshot):

      "Some nodes are still deleting bucket"

      Diagnostics are attached.

      1. 10.3.3.104-8091-diag.txt.gz
        1.79 MB
        Deepkaran Salooja
      2. 10.3.3.106-8091-diag.txt.gz
        2.84 MB
        Deepkaran Salooja
      3. 10.3.3.107-8091-diag.txt.gz
        1.03 MB
        Deepkaran Salooja
      4. 10.3.3.95-8091-diag.txt.gz
        2.52 MB
        Deepkaran Salooja
      1. BucketDelete.png
        130 kB
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Show
        andreibaranouski Andrei Baranouski added a comment - test: http://qa.hq.northscale.net/job/centos-64-2.0-view-query-extended-tests/70/consoleFull
        Hide
        chiyoung Chiyoung Seo added a comment -

        Mike,

        Please take a look at the diag files to see if this issue was caused by the ep-engine side. Note that there was a regression (MB-6711) from our recent changes in ep-engine, which caused the flusher to keep failed continuously. I'm not sure if the root cause of this issue is from MB-6711.

        Show
        chiyoung Chiyoung Seo added a comment - Mike, Please take a look at the diag files to see if this issue was caused by the ep-engine side. Note that there was a regression ( MB-6711 ) from our recent changes in ep-engine, which caused the flusher to keep failed continuously. I'm not sure if the root cause of this issue is from MB-6711 .
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Chiyoung
        is MB-67-8 also related to the same change set ?
        for today , to unblock QE I am locking down the ep-engine manifest to
        https://github.com/couchbase/ep-engine/commit/f0352955cd4f2b60352d1ba6bac0d44446b8882f

        we can unlock the manifest tomorrow to point to master/HEAD

        Show
        farshid Farshid Ghods (Inactive) added a comment - Chiyoung is MB-67-8 also related to the same change set ? for today , to unblock QE I am locking down the ep-engine manifest to https://github.com/couchbase/ep-engine/commit/f0352955cd4f2b60352d1ba6bac0d44446b8882f we can unlock the manifest tomorrow to point to master/HEAD
        Hide
        jin Jin Lim (Inactive) added a comment - - edited

        Fashid, assuming you meant MB-6708 (not MB-67-8) it looks like the same issue. Somehow EP Engine gets stuck in infinite notifying loop of new header position of a given vbucket.
        I am going to mark 6708 as a duplicate of MB6711 for now. BTW how this rebalance issue could have not revealed by running the testrunner's simple-test? We may want to ensure that such test case is part of the simple regression test. Thanks.

        Show
        jin Jin Lim (Inactive) added a comment - - edited Fashid, assuming you meant MB-6708 (not MB-67-8) it looks like the same issue. Somehow EP Engine gets stuck in infinite notifying loop of new header position of a given vbucket. I am going to mark 6708 as a duplicate of MB6711 for now. BTW how this rebalance issue could have not revealed by running the testrunner's simple-test? We may want to ensure that such test case is part of the simple regression test. Thanks.
        Hide
        jin Jin Lim (Inactive) added a comment -

        Sorry I meant to close 6708 instead of this. Thanks.

        Show
        jin Jin Lim (Inactive) added a comment - Sorry I meant to close 6708 instead of this. Thanks.
        Hide
        mikew Mike Wiederhold added a comment -

        http://review.couchbase.org/#/c/21069/1

        Note that creating and deleting a bucket very quickly might still cause this issue due to latencies between mccouch and ep-engine.

        Show
        mikew Mike Wiederhold added a comment - http://review.couchbase.org/#/c/21069/1 Note that creating and deleting a bucket very quickly might still cause this issue due to latencies between mccouch and ep-engine.
        Hide
        deepkaran.salooja Deepkaran Salooja added a comment -

        Verified with build 1820

        Show
        deepkaran.salooja Deepkaran Salooja added a comment - Verified with build 1820

          People

          • Assignee:
            mikew Mike Wiederhold
            Reporter:
            deepkaran.salooja Deepkaran Salooja
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes