Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42961

Undeploy/Pause closes source bkt reference even when there are other functions deployed against it

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.6.0, 6.6.1
    • Fix Version/s: 6.6.1
    • Component/s: eventing
    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.

      • System should have more than 1 functions deployed on the same src-bkt
      • KV rebalance should be triggered
      • As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt
      • If system hits this race condition, rebalance will be stuck and will fail after 10 mins.

      customer deployment ran into this issue twice.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            vikas.chaudhary Vikas Chaudhary added a comment - - edited

            verified on toy/6.6.1-9207

            Show
            vikas.chaudhary Vikas Chaudhary added a comment - - edited verified on toy/6.6.1-9207
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message:
            MB-42961: Refresh bucket object when its out of sync with the bucket uuid

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message: MB-42961 : Refresh bucket object when its out of sync with the bucket uuid
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message:
            MB-42961: Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message: MB-42961 : Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message:
            MB-42961: Unwatch bucket when all handlers listening on the bucket gets undeployed/paused

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message: MB-42961 : Unwatch bucket when all handlers listening on the bucket gets undeployed/paused
            Hide
            jeelan.poola Jeelan Poola added a comment -

            Thank you Wayne Siu!
            We shall target have the fix ready before RC2 is picked.

            Show
            jeelan.poola Jeelan Poola added a comment - Thank you Wayne Siu ! We shall target have the fix ready before RC2 is picked.
            Hide
            ritam.sharma Ritam Sharma added a comment -

            Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6

            Show
            ritam.sharma Ritam Sharma added a comment - Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6
            Hide
            jeelan.poola Jeelan Poola added a comment -

            Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you!

            Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.

            Show
            jeelan.poola Jeelan Poola added a comment - Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you! Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.

              People

              Assignee:
              ankit.prabhu Ankit Prabhu
              Reporter:
              ankit.prabhu Ankit Prabhu
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  PagerDuty