Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44799

Rebalance stopped by janitor

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • ns_server
    • Centos 7 64 bit; CB EE 7.0.0-4603

    Description

      Summary:
      Cloning off the logs from MB-44798. The issue here is to track as to why the janitor stopped the rebalance (4th step at MB-44798). (If its not a consequence of the issue at MB-44798)

      Few observations:
      on .215 error.log

      [ns_server:error,2021-03-08T00:55:06.259-08:00,ns_1@172.23.97.215:leader_quorum_nodes_manager<0.657.0>:ns_config_rep:synchronize_remote:306]Failed to synchronize config to some nodes: 
      [{'ns_1@172.23.107.5',timeout}]
      [ns_server:error,2021-03-08T01:01:35.541-08:00,ns_1@172.23.97.215:<0.26825.24>:ns_rebalance_observer:generic_get_call:114]Unexpected exception {exit,
                               {noproc,
                                   {gen_server,call,
                                       [{via,leader_registry,ns_rebalance_observer},
                                        get_aggregated_progress,10000]}}}
      [ns_server:error,2021-03-08T01:01:35.541-08:00,ns_1@172.23.97.215:<0.26825.24>:rebalance:progress:153]Couldn't reach ns_rebalance_observer
      [ns_server:error,2021-03-08T01:01:45.806-08:00,ns_1@172.23.97.215:<0.12379.24>:ns_rebalance_observer:generic_get_call:114]Unexpected exception {exit,
                               {noproc,
                                   {gen_server,call,
                                       [{via,leader_registry,ns_rebalance_observer},
                                        get_aggregated_progress,10000]}}}
      [ns_server:error,2021-03-08T01:01:45.806-08:00,ns_1@172.23.97.215:<0.12379.24>:rebalance:progress:153]Couldn't reach ns_rebalance_observer
      [ns_server:error,2021-03-08T01:01:56.069-08:00,ns_1@172.23.97.215:<0.4634.25>:ns_rebalance_observer:generic_get_call:114]Unexpected exception {exit,
                               {noproc,
                                   {gen_server,call,
                                       [{via,leader_registry,ns_rebalance_observer},
                                        get_aggregated_progress,10000]}}}
      [ns_server:error,2021-03-08T01:01:56.070-08:00,ns_1@172.23.97.215:<0.4634.25>:rebalance:progress:153]Couldn't reach ns_rebalance_observer
      [ns_server:error,2021-03-08T01:12:39.823-08:00,ns_1@172.23.97.215:<0.5000.27>:diag_handler:handle_diag_eval:785]WARNING: /diag/eval:

      Screenshots and logs attached. 

      CC: Umang Ritam Sharma

      Attachments

        1. rebalance_stopped_by_janitor.png
          rebalance_stopped_by_janitor.png
          455 kB
        2. resetting_rebalance.png
          resetting_rebalance.png
          491 kB
        3. screenshot-1.png
          screenshot-1.png
          34 kB
        4. screenshot-2.png
          screenshot-2.png
          41 kB
        5. screenshot-3.png
          screenshot-3.png
          36 kB
        6. screenshot-4.png
          screenshot-4.png
          58 kB
        7. starting_rebalance.png
          starting_rebalance.png
          551 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            sumedh.basarkod Sumedh Basarkod (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty