Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7891

[windows] a node became down during rebalance

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
    • Environment:

      Description

      test to reproduce:
      -t swaprebalance.SwapRebalanceFailedTests.test_failover_swap_rebalance,replica=2,num-buckets=2,num-swap=2,keys-count=1000000,swap-orchestrator=True

      2013-03-10 10:10:39.411 ns_orchestrator:4:info:message(ns_1@10.1.3.82) - Starting rebalance, KeepNodes = ['ns_1@10.1.3.83','ns_1@10.1.3.86',
      'ns_1@10.1.3.85'], EjectNodes = []

      2013-03-10 10:10:39.998 ns_cluster:1:info:message(ns_1@10.1.3.84) - Node 'ns_1@10.1.3.84' is leaving cluster.
      2013-03-10 10:10:40.005 ns_rebalancer:0:info:message(ns_1@10.1.3.82) - Started rebalancing bucket bucket-1
      2013-03-10 10:10:40.583 ns_node_disco:5:warning:node down(ns_1@10.1.3.85) - Node 'ns_1@10.1.3.85' saw that node 'ns_1@10.1.3.84' went down.
      2013-03-10 10:10:40.692 ns_memcached:1:info:message(ns_1@10.1.3.85) - Bucket "bucket-1" loaded on node 'ns_1@10.1.3.85' in 0 seconds.
      2013-03-10 10:10:41.068 ns_node_disco:5:warning:node down(ns_1@10.1.3.82) - Node 'ns_1@10.1.3.82' saw that node 'ns_1@10.1.3.84' went down.
      2013-03-10 10:10:42.583 ns_vbucket_mover:0:info:message(ns_1@10.1.3.82) - Bucket "bucket-1" rebalance does not seem to be swap rebalance
      2013-03-10 10:27:51.964 ns_node_disco:5:warning:node down(ns_1@10.1.3.83) - Node 'ns_1@10.1.3.83' saw that node 'ns_1@10.1.3.85' went down.
      2013-03-10 10:27:52.021 ns_node_disco:5:warning:node down(ns_1@10.1.3.86) - Node 'ns_1@10.1.3.86' saw that node 'ns_1@10.1.3.85' went down.
      2013-03-10 10:27:58.943 ns_node_disco:5:warning:node down(ns_1@10.1.3.82) - Node 'ns_1@10.1.3.82' saw that node 'ns_1@10.1.3.85' went down.
      2013-03-10 10:28:02.651 ns_node_disco:4:info:node up(ns_1@10.1.3.83) - Node 'ns_1@10.1.3.83' saw that node 'ns_1@10.1.3.85' came up.
      2013-03-10 10:28:02.896 ns_node_disco:4:info:node up(ns_1@10.1.3.86) - Node 'ns_1@10.1.3.86' saw that node 'ns_1@10.1.3.85' came up.
      2013-03-10 10:28:03.990 ns_orchestrator:2:info:message(ns_1@10.1.3.82) - Rebalance exited with reason {{bulk_set_vbucket_state_failed,
      [{'ns_1@10.1.3.85',
      {'EXIT',
      {{nodedown,'ns_1@10.1.3.85'},
      {gen_server,call,
      [

      {'janitor_agent-bucket-1', 'ns_1@10.1.3.85'}

      ,
      {if_rebalance,<0.11567.86>,
      {update_vbucket_state,315,replica,
      undefined,'ns_1@10.1.3.83'}},
      infinity]}}}}]},
      [

      {janitor_agent,bulk_set_vbucket_state,4}

      ,

      {ns_vbucket_mover, update_replication_post_move,3}

      ,

      {ns_vbucket_mover,on_move_done,2}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}

      attaching logs

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Show
          iryna iryna added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7891/c431f173/10.1.3.83-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7891/c431f173/10.1.3.85-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7891/c431f173/10.1.3.86-diag.zip
          Hide
          farshid Farshid Ghods (Inactive) added a comment -

          Iryna

          please assign memcached crash and other windows related issues to Siri

          Show
          farshid Farshid Ghods (Inactive) added a comment - Iryna please assign memcached crash and other windows related issues to Siri
          Hide
          siri Sriram Melkote added a comment -

          This is tracked in MB-7902, where nodes disconnect without an underlying connectivity or node reachability issue

          Show
          siri Sriram Melkote added a comment - This is tracked in MB-7902 , where nodes disconnect without an underlying connectivity or node reachability issue
          Hide
          maria Maria McDuff (Inactive) added a comment -
          Show
          maria Maria McDuff (Inactive) added a comment - MB-7902

            People

            • Assignee:
              siri Sriram Melkote
              Reporter:
              iryna iryna
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes