Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26859

Disable lifecycle operations during rebalance and vice versa

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.5.0
    • 5.5.0
    • eventing

    Description

      Currently when we do a rebalance during bootstrapping state, rebalance hangs.

      As discussed in today's meeting we need following things to be implemented

      1) Disable eventing lifecycle operations like deploy/undeploy/pause/resume/import/delete/add during rebalance.

      2) If eventing lifecycle operations like deploy/undeploy/pause/resume/import/delete/add are on going, eventing node rebalance should fail.

      3) Delete should be available at all times to allow stopping and cleaning up any hung or failed state event handlers

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-26859
          # Subject Branch Project Status CR V

          Activity

            Build couchbase-server-5.5.0-2176 contains eventing commit 219d6541150c08c50f10dbb34e3320a53cd7a579 with commit message:
            MB-26859 Fail rebalance request if some apps are undergoing bootstrap
            https://github.com/couchbase/eventing/commit/219d6541150c08c50f10dbb34e3320a53cd7a579

            build-team Couchbase Build Team added a comment - Build couchbase-server-5.5.0-2176 contains eventing commit 219d6541150c08c50f10dbb34e3320a53cd7a579 with commit message: MB-26859 Fail rebalance request if some apps are undergoing bootstrap https://github.com/couchbase/eventing/commit/219d6541150c08c50f10dbb34e3320a53cd7a579

            Showed to Bala that Eventing might not receive notification about rebalance at the same time as KV. ns_server first performs KV+Views rebalance and after it finishes KV+views rebalance, MDS roles are notified about rebalance. Current testrunner case seem to wait till rebalance progress to go up to 10% and then it issues deploy request - at this time Eventing might not have received rebalance notification from ns_server and it would happily accept deployment request.

            So based on above observation some additional test tuning might be necessary on testrunner side.

            asingh Abhishek Singh (Inactive) added a comment - Showed to Bala that Eventing might not receive notification about rebalance at the same time as KV. ns_server first performs KV+Views rebalance and after it finishes KV+views rebalance, MDS roles are notified about rebalance. Current testrunner case seem to wait till rebalance progress to go up to 10% and then it issues deploy request - at this time Eventing might not have received rebalance notification from ns_server and it would happily accept deployment request. So based on above observation some additional test tuning might be necessary on testrunner side.

            Closing this out as code changes are on master. Some changes are likely needed in testrunner cases around this scenario.

            asingh Abhishek Singh (Inactive) added a comment - Closing this out as code changes are on master. Some changes are likely needed in testrunner cases around this scenario.

            Build couchbase-server-5.5.0-2208 contains eventing commit b80465ace55032c50055e7c0d892e194ba56adaa with commit message:
            MB-26859 Don't exit metakv callback functions on Eventing nodes...
            https://github.com/couchbase/eventing/commit/b80465ace55032c50055e7c0d892e194ba56adaa

            build-team Couchbase Build Team added a comment - Build couchbase-server-5.5.0-2208 contains eventing commit b80465ace55032c50055e7c0d892e194ba56adaa with commit message: MB-26859 Don't exit metakv callback functions on Eventing nodes... https://github.com/couchbase/eventing/commit/b80465ace55032c50055e7c0d892e194ba56adaa

            Validated this on 5.5.0-2637.

            Rebalance fails when eventing lifecycle operation is going on and vice versa.
            Logs : http://qa.sc.couchbase.com/job/temp_rebalance_even/279/console

            Hence marking this bug closed.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Validated this on 5.5.0-2637. Rebalance fails when eventing lifecycle operation is going on and vice versa. Logs : http://qa.sc.couchbase.com/job/temp_rebalance_even/279/console Hence marking this bug closed.

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty