Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40636

[BP MB-40300] - A rebalance is needed post eventing node failover to transfer VB ownership to active nodes

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Cheshire-Cat, 6.0.4, 6.5.1, 6.6.0
    • Fix Version/s: 6.6.1
    • Component/s: eventing
    • Story Points:
      1

      Description

      Although this behaviour is as per design, this ticket is to investigate if things can be improved in fail-over scenario. There may be some complications in forcefully taking over VB ownership without a rebalance. Especially if recovery is performed soon after marking a node failed-over (without rebalance).

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.0-7913 contains eventing commit b6c8230 with commit message:
            MB-40636 : Rename ERR_REBALANCE_ONGOING to ERR_REBALANCE_OR_FAILOVER_ONGOING

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7913 contains eventing commit b6c8230 with commit message: MB-40636 : Rename ERR_REBALANCE_ONGOING to ERR_REBALANCE_OR_FAILOVER_ONGOING
            Hide
            ritam.sharma Ritam Sharma added a comment -

            Jeelan Poola - I updated the ticket from bug to improvement. The current behaviour is as per design, this code change is to make the failover handling better.

            Show
            ritam.sharma Ritam Sharma added a comment - Jeelan Poola - I updated the ticket from bug to improvement. The current behaviour is as per design, this code change is to make the failover handling better.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.0-7915 contains eventing commit e586775 with commit message:
            MB-40636 : turn on auto-redistribute-vbs-on-failover by default

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7915 contains eventing commit e586775 with commit message: MB-40636 : turn on auto-redistribute-vbs-on-failover by default
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9013 contains eventing commit e586775 with commit message:
            MB-40636 : turn on auto-redistribute-vbs-on-failover by default

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9013 contains eventing commit e586775 with commit message: MB-40636 : turn on auto-redistribute-vbs-on-failover by default
            Hide
            vikas.chaudhary Vikas Chaudhary added a comment -

            Automated the following cases

             

            Verify getAggRebalanceStatus is true when failover is running

            Disable the setting and verify data loss

            Failover only

            Failover 1 eventing node with bucket op /timer/curl/n1ql

            Failover 1 eventing node when one failover is already running in background

            Failover 2 eventing node

            Hard/Graceful Failover 1 kv node

            Hard/Graceful Failover 2 kv nodes

            Hard/Graceful Failover node kv and eventing service on same node

            Failover and Add back

            Failover 1 eventing node with bucket op /timer/curl/n1ql

            Failover 1 eventing node when one failover is already running in background

            Failover 2 eventing node

            Failover 1 kv node

            Failover 2 kv nodes

            Failover node kv and eventing service on same node

            Failover + Add back  and rebalance

            Failover 1 eventing node with bucket op /timer/curl/n1ql

            Failover 1 eventing node when one failover is already running in background

            Failover 2 eventing node

            Failover 1 kv node

            Failover 2 kv nodes

            Failover node kv and eventing service on same node

            Failover + Rebalance out

            Failover 1 eventing node with bucket op /timer/curl/n1ql

            Failover 1 eventing node when one failover is already running in background

            Failover 2 eventing node

            Failover 1 kv node

            Failover 2 kv nodes

            Failover node kv and eventing service on same node

            Multihandler + Failover

            Mix of pause and deploy handler + Failover

            VB_shuffle in progress and trigger rebalance (expected to fail)

            VB_shuffle in progress and trigger lifecycle operations (expected to fail)

            Show
            vikas.chaudhary Vikas Chaudhary added a comment - Automated the following cases   Verify getAggRebalanceStatus is true when failover is running Disable the setting and verify data loss Failover only Failover 1 eventing node with bucket op /timer/curl/n1ql Failover 1 eventing node when one failover is already running in background Failover 2 eventing node Hard/Graceful Failover 1 kv node Hard/Graceful Failover 2 kv nodes Hard/Graceful Failover node kv and eventing service on same node Failover and Add back Failover 1 eventing node with bucket op /timer/curl/n1ql Failover 1 eventing node when one failover is already running in background Failover 2 eventing node Failover 1 kv node Failover 2 kv nodes Failover node kv and eventing service on same node Failover + Add back  and rebalance Failover 1 eventing node with bucket op /timer/curl/n1ql Failover 1 eventing node when one failover is already running in background Failover 2 eventing node Failover 1 kv node Failover 2 kv nodes Failover node kv and eventing service on same node Failover + Rebalance out Failover 1 eventing node with bucket op /timer/curl/n1ql Failover 1 eventing node when one failover is already running in background Failover 2 eventing node Failover 1 kv node Failover 2 kv nodes Failover node kv and eventing service on same node Multihandler + Failover Mix of pause and deploy handler + Failover VB_shuffle in progress and trigger rebalance (expected to fail) VB_shuffle in progress and trigger lifecycle operations (expected to fail)

              People

              Assignee:
              vikas.chaudhary Vikas Chaudhary
              Reporter:
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  PagerDuty