Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42961

Undeploy/Pause closes source bkt reference even when there are other functions deployed against it

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.6.0, 6.6.1
    • 6.6.1
    • eventing
    • Untriaged
    • 1
    • Unknown

    Description

      This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.

      • System should have more than 1 functions deployed on the same src-bkt
      • KV rebalance should be triggered
      • As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt
      • If system hits this race condition, rebalance will be stuck and will fail after 10 mins.

      customer deployment ran into this issue twice.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you!

            Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.

            jeelan.poola Jeelan Poola added a comment - Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you! Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.
            ritam.sharma Ritam Sharma added a comment -

            Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6

            ritam.sharma Ritam Sharma added a comment - Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6
            jeelan.poola Jeelan Poola added a comment -

            Thank you Wayne Siu!
            We shall target have the fix ready before RC2 is picked.

            jeelan.poola Jeelan Poola added a comment - Thank you Wayne Siu ! We shall target have the fix ready before RC2 is picked.

            Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message:
            MB-42961: Unwatch bucket when all handlers listening on the bucket gets undeployed/paused

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message: MB-42961 : Unwatch bucket when all handlers listening on the bucket gets undeployed/paused

            Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message:
            MB-42961: Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message: MB-42961 : Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt

            Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message:
            MB-42961: Refresh bucket object when its out of sync with the bucket uuid

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message: MB-42961 : Refresh bucket object when its out of sync with the bucket uuid
            vikas.chaudhary Vikas Chaudhary added a comment - - edited

            verified on toy/6.6.1-9207

            vikas.chaudhary Vikas Chaudhary added a comment - - edited verified on toy/6.6.1-9207

            People

              ankit.prabhu Ankit Prabhu
              ankit.prabhu Ankit Prabhu
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty