Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62725

[Rebalance][K8S] : Cluster status changes from unbalanced to balanced suddenly post rebalance failure

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Created a 3 node cluster on k8s with operator with all services
      2. On one pod, memcached was killed in a loop. Multiple failovers and rebalance failures occur as expected
      3. Stopped the memcached kill loop.
      4. Rebalances beyond this fail in a loop(as triggered by the operator again and again) - tracked in MB-62724.
      5. Rebalance fails with leader_activities_error.
      6. Cluster status suddenly changes from unbalanced to balanced post this failure

      Rebalance exited with reason {{badmatch,
      		 
      {leader_activities_error,
      		 
      {default,rebalance},
      		 
      {quorum_lost,
      		 
      {lease_lost,
      		 
      'ns_1@cb-example-0001.cb-example.default.svc'}}}},
      		 
      [{ns_rebalancer,rebalance,7,
      		 
      [{file,"src/ns_rebalancer.erl"},{line,456}]},
      		 
      {proc_lib,init_p_do_apply,3,
      		 
      [{file,"proc_lib.erl"},{line,240}]}]}.
      		 
      Rebalance Operation Id = 275e6c370d3c4bac4f45fe2fc175764b

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            raghav.sk Raghav S K
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty