Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60784

[Upgrade][FTS] : Rebalance exited with reason {{badmatch,{leader_activities_error,{default,rebalance},{quorum_lost,{lease_lost,'ns_1@172.23.104.66'}}}},[{ns_rebalancer,rebalance,7,[{file,"src/ns_rebalancer.erl"},{line,456}]},

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Critical
    • 7.6.0
    • 7.6.0
    • fts, ns_server
    • Operating System : Operating System : Debian GNU/Linux 11 (bullseye)
      Initial Version : Couchbase Enterprise Edition 7.2.4-7070
      Upgrade Version : Couchbase Enterprise Edition 7.6.0-2133

    Description

      Steps to reproduce

      1. Created a couchbase cluster with the following configurations
        1. 172.23.104.66 - kv
        2. 172.23.105.179 - kv, n1ql
        3. 172.23.105.192 - fts
        4. 172.23.105.199 - fts
        5. 172.23.105.202 - fts
      2. Created a magma bucket named "test_bucket" and loaded 10k documents onto it
      3. Throughout the process of upgrades, there was constant mutations 
      4. Created a search index - definition attached
      5. Throughout the process of upgrades, there was search queries
      6. Upgraded node 172.23.105.192 by failing over, upgrading and adding it back
      7. Upgraded node 172.23.105.179 in a similar manner
      8. Upgraded node 172.23.105.199 in a similar manner
      9. Upgraded node 172.23.105.202 in a similar manner

      Rebalance fails

      Rebalance exited with reason {{badmatch,
      		 
      {leader_activities_error,
      		 
      {default,rebalance},
      		 
      {quorum_lost,
      		 
      {lease_lost,'ns_1@172.23.104.66'}}}},
      		 
      [{ns_rebalancer,rebalance,7,
      		 
      [{file,"src/ns_rebalancer.erl"},{line,456}]},
      		 
      {proc_lib,init_p_do_apply,3,
      		 
      [{file,"proc_lib.erl"},{line,240}]}]}.
      		 
      Rebalance Operation Id = 714c0fc89e4b6049c2760a2bde8e86b4

      Retried the rebalance multiple times and it always fails, hence marking this as a blocker

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            raghav.sk Raghav S K
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty