Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7891

[windows] a node became down during rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 2.1.0
    • 2.0.1
    • couchbase-bucket
    • Security Level: Public

    Description

      test to reproduce:
      -t swaprebalance.SwapRebalanceFailedTests.test_failover_swap_rebalance,replica=2,num-buckets=2,num-swap=2,keys-count=1000000,swap-orchestrator=True

      2013-03-10 10:10:39.411 ns_orchestrator:4:info:message(ns_1@10.1.3.82) - Starting rebalance, KeepNodes = ['ns_1@10.1.3.83','ns_1@10.1.3.86',
      'ns_1@10.1.3.85'], EjectNodes = []

      2013-03-10 10:10:39.998 ns_cluster:1:info:message(ns_1@10.1.3.84) - Node 'ns_1@10.1.3.84' is leaving cluster.
      2013-03-10 10:10:40.005 ns_rebalancer:0:info:message(ns_1@10.1.3.82) - Started rebalancing bucket bucket-1
      2013-03-10 10:10:40.583 ns_node_disco:5:warning:node down(ns_1@10.1.3.85) - Node 'ns_1@10.1.3.85' saw that node 'ns_1@10.1.3.84' went down.
      2013-03-10 10:10:40.692 ns_memcached:1:info:message(ns_1@10.1.3.85) - Bucket "bucket-1" loaded on node 'ns_1@10.1.3.85' in 0 seconds.
      2013-03-10 10:10:41.068 ns_node_disco:5:warning:node down(ns_1@10.1.3.82) - Node 'ns_1@10.1.3.82' saw that node 'ns_1@10.1.3.84' went down.
      2013-03-10 10:10:42.583 ns_vbucket_mover:0:info:message(ns_1@10.1.3.82) - Bucket "bucket-1" rebalance does not seem to be swap rebalance
      2013-03-10 10:27:51.964 ns_node_disco:5:warning:node down(ns_1@10.1.3.83) - Node 'ns_1@10.1.3.83' saw that node 'ns_1@10.1.3.85' went down.
      2013-03-10 10:27:52.021 ns_node_disco:5:warning:node down(ns_1@10.1.3.86) - Node 'ns_1@10.1.3.86' saw that node 'ns_1@10.1.3.85' went down.
      2013-03-10 10:27:58.943 ns_node_disco:5:warning:node down(ns_1@10.1.3.82) - Node 'ns_1@10.1.3.82' saw that node 'ns_1@10.1.3.85' went down.
      2013-03-10 10:28:02.651 ns_node_disco:4:info:node up(ns_1@10.1.3.83) - Node 'ns_1@10.1.3.83' saw that node 'ns_1@10.1.3.85' came up.
      2013-03-10 10:28:02.896 ns_node_disco:4:info:node up(ns_1@10.1.3.86) - Node 'ns_1@10.1.3.86' saw that node 'ns_1@10.1.3.85' came up.
      2013-03-10 10:28:03.990 ns_orchestrator:2:info:message(ns_1@10.1.3.82) - Rebalance exited with reason {{bulk_set_vbucket_state_failed,
      [{'ns_1@10.1.3.85',
      {'EXIT',
      {{nodedown,'ns_1@10.1.3.85'},
      {gen_server,call,
      [

      {'janitor_agent-bucket-1', 'ns_1@10.1.3.85'}

      ,
      {if_rebalance,<0.11567.86>,
      {update_vbucket_state,315,replica,
      undefined,'ns_1@10.1.3.83'}},
      infinity]}}}}]},
      [

      {janitor_agent,bulk_set_vbucket_state,4}

      ,

      {ns_vbucket_mover, update_replication_post_move,3}

      ,

      {ns_vbucket_mover,on_move_done,2}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}

      attaching logs

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-7902 [windows] Rebalance exited with reason {detected_nodes_change, {ns_node_disco_events,

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              siri Sriram Melkote (Inactive)
              iryna iryna
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty