Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42961

Undeploy/Pause closes source bkt reference even when there are other functions deployed against it

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.6.0, 6.6.1
    • Fix Version/s: 6.6.1
    • Component/s: eventing
    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.

      • System should have more than 1 functions deployed on the same src-bkt
      • KV rebalance should be triggered
      • As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt
      • If system hits this race condition, rebalance will be stuck and will fail after 10 mins.

      customer deployment ran into this issue twice.

        Attachments

          Issue Links

          For Gerrit Dashboard: MB-42961
          # Subject Branch Project Status CR V

            Activity

            Hide
            jeelan.poola Jeelan Poola added a comment -

            Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you!

            Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.

            Show
            jeelan.poola Jeelan Poola added a comment - Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you! Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.
            Hide
            ritam.sharma Ritam Sharma added a comment -

            Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6

            Show
            ritam.sharma Ritam Sharma added a comment - Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6
            Hide
            jeelan.poola Jeelan Poola added a comment -

            Thank you Wayne Siu!
            We shall target have the fix ready before RC2 is picked.

            Show
            jeelan.poola Jeelan Poola added a comment - Thank you Wayne Siu ! We shall target have the fix ready before RC2 is picked.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message:
            MB-42961: Unwatch bucket when all handlers listening on the bucket gets undeployed/paused

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message: MB-42961 : Unwatch bucket when all handlers listening on the bucket gets undeployed/paused
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message:
            MB-42961: Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message: MB-42961 : Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message:
            MB-42961: Refresh bucket object when its out of sync with the bucket uuid

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message: MB-42961 : Refresh bucket object when its out of sync with the bucket uuid
            Hide
            vikas.chaudhary Vikas Chaudhary added a comment - - edited

            verified on toy/6.6.1-9207

            Show
            vikas.chaudhary Vikas Chaudhary added a comment - - edited verified on toy/6.6.1-9207

              People

              Assignee:
              ankit.prabhu Ankit Prabhu
              Reporter:
              ankit.prabhu Ankit Prabhu
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  PagerDuty