Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60784

[Upgrade][FTS] : Rebalance exited with reason {{badmatch,{leader_activities_error,{default,rebalance},{quorum_lost,{lease_lost,'ns_1@172.23.104.66'}}}},[{ns_rebalancer,rebalance,7,[{file,"src/ns_rebalancer.erl"},{line,456}]},

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Critical
    • 7.6.0
    • 7.6.0
    • fts, ns_server
    • Operating System : Operating System : Debian GNU/Linux 11 (bullseye)
      Initial Version : Couchbase Enterprise Edition 7.2.4-7070
      Upgrade Version : Couchbase Enterprise Edition 7.6.0-2133

    Description

      Steps to reproduce

      1. Created a couchbase cluster with the following configurations
        1. 172.23.104.66 - kv
        2. 172.23.105.179 - kv, n1ql
        3. 172.23.105.192 - fts
        4. 172.23.105.199 - fts
        5. 172.23.105.202 - fts
      2. Created a magma bucket named "test_bucket" and loaded 10k documents onto it
      3. Throughout the process of upgrades, there was constant mutations 
      4. Created a search index - definition attached
      5. Throughout the process of upgrades, there was search queries
      6. Upgraded node 172.23.105.192 by failing over, upgrading and adding it back
      7. Upgraded node 172.23.105.179 in a similar manner
      8. Upgraded node 172.23.105.199 in a similar manner
      9. Upgraded node 172.23.105.202 in a similar manner

      Rebalance fails

      Rebalance exited with reason {{badmatch,
      		 
      {leader_activities_error,
      		 
      {default,rebalance},
      		 
      {quorum_lost,
      		 
      {lease_lost,'ns_1@172.23.104.66'}}}},
      		 
      [{ns_rebalancer,rebalance,7,
      		 
      [{file,"src/ns_rebalancer.erl"},{line,456}]},
      		 
      {proc_lib,init_p_do_apply,3,
      		 
      [{file,"proc_lib.erl"},{line,240}]}]}.
      		 
      Rebalance Operation Id = 714c0fc89e4b6049c2760a2bde8e86b4

      Retried the rebalance multiple times and it always fails, hence marking this as a blocker

      Attachments

        1. Screenshot 2024-02-14 at 1.02.10 PM.png
          614 kB
          Bryan McCoid
        2. Screenshot 2024-02-14 at 1.02.30 PM.png
          620 kB
          Bryan McCoid
        3. Screenshot 2024-02-14 at 1.03.04 PM.png
          437 kB
          Bryan McCoid
        4. Screenshot 2024-02-14 at 1.03.17 PM.png
          439 kB
          Bryan McCoid
        5. Screenshot 2024-02-14 at 1.14.58 PM.png
          731 kB
          Bryan McCoid
        6. Screenshot 2024-02-14 at 1.17.09 PM.png
          727 kB
          Bryan McCoid
        7. Screenshot 2024-02-14 at 2.33.39 PM.png
          311 kB
          Abhi Dangeti
        8. Screenshot 2024-02-14 at 2.38.52 PM.png
          206 kB
          Abhi Dangeti
        9. Screenshot 2024-02-14 at 2.41.47 PM.png
          200 kB
          Abhi Dangeti
        10. Screenshot 2024-02-14 at 2.42.20 PM.png
          384 kB
          Abhi Dangeti
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            raghav.sk Raghav S K
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty