Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60002

[Rebalance][Windows] : Delta Recovery rebalance fails with Rebalance exited with reason {{badmatch, {error,marking_as_warmed_failed, ['ns_1@172.23.136.134']}}, [{ns_rebalancer, '-apply_delta_recovery_buckets/3-fun-0-',2, [{file,"src/ns_rebalancer.erl"},

    XMLWordPrintable

Details

    Description

      Steps to repro

      1. Created a 2 node cluster using Windows Server 2019 nodes on 7.2.0-5325
      2. Loaded sample buckets and created a few buckets and loaded data
      3. Upgraded the cluster to 7.2.3-6710 and continued data load during upgrade
      4. Post upgrade performed a graceful failover of one of the nodes
      5. Added it back with Delta recovery
      6. Rebalance fails. A re-try of the rebalance succeeds
      7. Failed the node over again and retried delta recovery, it succeeds

      Rebalance fails 

      Rebalance exited with reason {{badmatch,
      {error,marking_as_warmed_failed,
      ['ns_1@172.23.136.134']}},
      [{ns_rebalancer,
      '-apply_delta_recovery_buckets/3-fun-0-',2,
      [{file,"src/ns_rebalancer.erl"},
      {line,1038}]},
      {lists,foreach,2,
      [{file,"lists.erl"},{line,1342}]},
      {ns_rebalancer,apply_delta_recovery_buckets,3,
      [{file,"src/ns_rebalancer.erl"},
      {line,1035}]},
      {ns_rebalancer,rebalance_body,5,
      [{file,"src/ns_rebalancer.erl"},
      {line,501}]},
      {async,'-async_init/4-fun-1-',3,
      [{file,"src/async.erl"},{line,191}]}]}.
      Rebalance Operation Id = 07a9d32a3fc2b9b457ccbc00ce49266d 

       

      Tried to reproduce with the same cluster setup and the issue is not consistent ( as seen in the logs )

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            raghav.sk Raghav S K
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty