Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25238

Rebalance fails when adding back a failed over node with delta recovery

    XMLWordPrintable

Details

    Description

      Build : 4.6.3-4059

      This issue is seen often in a system test which scales upto 100M+ items. There is a step to fail over a KV node and then add it back to the cluster with delta recovery. This rebalance sometimes fails.

      [user:info,2017-07-13T08:18:48.237Z,ns_1@172.31.2.128:<0.980.0>:ns_orchestrator:handle_info:525]Rebalance exited with reason

      {badmatch,fail}

      [error_logger:error,2017-07-13T08:18:48.237Z,ns_1@172.31.2.128:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
      =========================CRASH REPORT=========================
      crasher:
      initial call: erlang:apply/2
      pid: <0.31843.10>
      registered_name: []
      exception error: no match of right hand side value fail
      in function ns_rebalancer:'apply_delta_recovery_buckets/3-fun-0'/2 (src/ns_rebalancer.erl, line 1122)
      in call from lists:foreach/2 (lists.erl, line 1323)
      in call from ns_rebalancer:apply_delta_recovery_buckets/3 (src/ns_rebalancer.erl, line 1120)
      in call from ns_rebalancer:rebalance/6 (src/ns_rebalancer.erl, line 598)
      ancestors: [<0.980.0>,ns_orchestrator_sup,mb_master_sup,mb_master,
      <0.812.0>,ns_server_sup,ns_server_nodes_sup,<0.166.0>,
      ns_server_cluster_sup,<0.89.0>]
      messages: []
      links: [<0.980.0>]
      dictionary: []
      trap_exit: false
      status: running
      heap_size: 75113
      stack_size: 27
      reductions: 376473
      neighbours:

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-25238
          # Subject Branch Project Status CR V

          Activity

            People

              ajit.yagaty Ajit Yagaty [X] (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty