Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54704

Probable rebalance problem while server upgrade from 6.6.x/7.0.x to 7.1.x

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 7.1.2, 7.1.3
    • ns_server
    • None
    • Untriaged
    • 0
    • Unknown

    Description

      Summary:

      While upgrading a 3 node cluster from server version 6.6.4/6.6.5/7.0.4/7.0.5 to 7.1.2/7.1.3 using CAO, though the upgrade seems to be successful, there are some intermittent rebalance issue where the server continuously responds with 

      "balanced": false
      

      This does not happen when upgrading from 6.6.x to 7.0.x.

       

      Details:

      • Scenario: While upgrading from 6.6.5 to 7.1.2 using CAO
      • Activity: looking into error.log file for one of the nodes(cb-example-0003), I could see these error as below:

       

                1.  Refusing to downgrade the compat version from [6,6] to undefined.

       ns_server:error,2022-11-29T13:33:57.689Z,ns_1@cb-example-0003.cb-example.default.svc:compat_mode_manager<0.2122.0>:cluster_compat_mode:do_consider_switching_compat_mode:282]Refusing to downgrade the compat version from [6,6] to undefined.

             Where, it seems it can't understand AnotherVersion from the code below:

              https://github.com/couchbase/ns_server/blob/d1b77cc2910d1ae06c6c89b0c14779511bfecc33/src/cluster_compat_mode.erl#L295-L301

       

              2. Unexpected exception

      [ns_server:error,2022-11-29T13:33:36.148Z,ns_1@cb-example-0002.cb-example.default.svc:<0.2232.0>:ns_rebalance_observer:generic_get_call:110]Unexpected exception {exit,                         {noproc,                             {gen_server,call,                                 [{via,leader_registry,ns_rebalance_observer},                                  get_detailed_progress,10000]}}}[ns_server:error,2022-11-29T13:33:36.151Z,ns_1@cb-example-0002.cb-example.default.svc:<0.2232.0>:ns_rebalance_observer:generic_get_call:110]Unexpected exception {exit,                         {noproc,                             {gen_server,call,                                 [{via,leader_registry,ns_rebalance_observer},                                  {get_rebalance_info,[]},                                  10000]}}} 

       

            Attaching the error.log files for all 6 nodes for upgrade from 6.6.5 to 7.1.2.

       

      P.S. Another observation, which is a bit intermittent - if you have only data service, this above error doesn't come and rebalance works fine.

      Attachments

        1. cb-example-0000.log
          7 kB
          Abhi Bose
        2. cb-example-0001.log
          8 kB
          Abhi Bose
        3. cb-example-0002.log
          8 kB
          Abhi Bose
        4. cb-example-0003.log
          50 kB
          Abhi Bose
        5. cb-example-0004.log
          95 kB
          Abhi Bose
        6. cb-example-0005.log
          95 kB
          Abhi Bose

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Abhijeeth.Nuthan Abhijeeth Nuthan
              abhi.bose Abhi Bose (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty