Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15652

[system tests windows] "Rebalance exited with reason {unexpected_exit," after "Control connection to memcached on 'ns_1@172.23.105.167' disconnected"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 4.0.0
    • 4.0.0
    • memcached
    • Security Level: Public
    • 4.0.0-3402
    • Untriaged
    • Windows 64-bit
    • Unknown

    Description

      steps:
      1. 3 nodes in cluster, 4 buckets. run data loader more then a day
      2. setup replication from SRC to cluster DEST for all buckets.
      3. rebalance in at SRC cluster
      rebalance in 2+2 nodes at DEST cluster

      result on dest cluster:

      Control connection to memcached on 'ns_1@172.23.105.167' disconnected: {{badmatch,
      {error,
      timeout}},
      [{mc_client_binary,
      cmd_vocal_recv,
      5,
      [

      {file, "src/mc_client_binary.erl"}, {line, 156}]},
      {mc_client_binary,
      select_bucket,
      2,
      [{file, "src/mc_client_binary.erl"}

      ,

      {line, 351}]},
      {ns_memcached,
      ensure_bucket,
      2,
      [{file, "src/ns_memcached.erl"}, {line, 1292}]},
      {ns_memcached,
      handle_info,
      2,
      [{file, "src/ns_memcached.erl"}, {line, 745}]},
      {gen_server,
      handle_msg,
      5,
      [{file, "gen_server.erl"}, {line, 604}]},
      {ns_memcached,
      init,
      1,
      [{file, "src/ns_memcached.erl"}, {line, 174}]},
      {gen_server,
      init_it,
      6,
      [{file, "gen_server.erl"}, {line, 304}]},
      {proc_lib,
      init_p_do_apply,
      3,
      [{file, "proc_lib.erl"}, {line, 239}]}]} (repeated 3 times) ns_memcached000 ns_1@172.23.105.167 07:45:00 - Mon Jul 13, 2015
      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.11817.17>,
      {{badmatch,{error,timeout,
      [{mc_client_binary,cmd_vocal_recv,5,
      [{file,"src/mc_client_binary.erl"}, {line,156}]},
      {mc_client_binary,select_bucket,2,
      [{file,"src/mc_client_binary.erl"}, {line,351}]},
      {ns_memcached,ensure_bucket,2,
      [{file,"src/ns_memcached.erl"}, {line,1292}]},
      {ns_memcached,handle_info,2,
      [{file,"src/ns_memcached.erl"}, {line,745}]},
      {gen_server,handle_msg,5,
      [{file,"gen_server.erl"},{line,604}]},
      {ns_memcached,init,1,
      [{file,"src/ns_memcached.erl"}, {line,174}]},
      {gen_server,init_it,6,
      [{file,"gen_server.erl"},{line,304}]
      },
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,239}]}]},
      {gen_server,call,
      ['ns_memcached-UserInfo', {delete_vbucket,437},
      360000]}},
      {gen_server,call,
      [{'janitor_agent-UserInfo', 'ns_1@172.23.105.167'},
      {if_rebalance,<0.30859.16>,
      {update_vbucket_state,433,active,paused,
      undefined}},
      infinity]}}}}
      ns_orchestrator002 ns_1@172.23.105.167 07:43:48 - Mon Jul 13, 2015
      <0.11616.17> exited with {unexpected_exit,
      {'EXIT',<0.11817.17>,
      {{badmatch,{error,timeout,
      [{mc_client_binary,cmd_vocal_recv,5,
      [{file,"src/mc_client_binary.erl"}, {line,156}]},
      {mc_client_binary,select_bucket,2,
      [{file,"src/mc_client_binary.erl"},{line,351}]},
      {ns_memcached,ensure_bucket,2,
      [{file,"src/ns_memcached.erl"},{line,1292}]},
      {ns_memcached,handle_info,2,
      [{file,"src/ns_memcached.erl"},{line,745}]
      },
      {gen_server,handle_msg,5,
      [{file,"gen_server.erl"},{line,604}]},
      {ns_memcached,init,1,
      [{file,"src/ns_memcached.erl"},{line,174}]},
      {gen_server,init_it,6,
      [{file,"gen_server.erl"},{line,304}]
      },
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,239}]}]},
      {gen_server,call,
      ['ns_memcached-UserInfo', {delete_vbucket,437},
      360000]}},
      {gen_server,call,
      [{'janitor_agent-UserInfo','ns_1@172.23.105.167'},
      {if_rebalance,<0.30859.16>,
      {update_vbucket_state,433,active,paused,
      undefined}},
      infinity]}}}} ns_vbucket_mover000 ns_1@172.23.105.167 07:43:48 - Mon Jul 13, 2015
      Control connection to memcached on 'ns_1@172.23.105.167' disconnected: {{badmatch,
      {error,
      timeout}},
      [{mc_client_binary,
      cmd_vocal_recv,
      5,
      [{file, "src/mc_client_binary.erl"}, {line, 156}]},
      {mc_client_binary,
      select_bucket,
      2,
      [{file, "src/mc_client_binary.erl"},{line, 351}

      ]},
      {ns_memcached,
      ensure_bucket,
      2,
      [

      {file, "src/ns_memcached.erl"}, {line, 1292}]},
      {ns_memcached,
      handle_info,
      2,
      [{file, "src/ns_memcached.erl"}

      ,

      {line, 745}

      ]},
      {gen_server,
      handle_msg,
      5,
      [

      {file, "gen_server.erl"}, {line, 604}]},
      {ns_memcached,
      init,
      1,
      [{file, "src/ns_memcached.erl"}, {line, 174}]},
      {gen_server,
      init_it,
      6,
      [{file, "gen_server.erl"}

      ,

      {line, 304}

      ]},
      {proc_lib,
      init_p_do_apply,
      3,
      [

      {file, "proc_lib.erl"}

      ,

      {line, 239}

      ]}]} ns_memcached000 ns_1@172.23.105.167 07:43:47 - Mon Jul 13, 2015
      Bucket "UserInfo" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.167 07:39:43 - Mon Jul 13, 2015
      Bucket "UserInfo" loaded on node 'ns_1@172.23.105.169' in 0 seconds. ns_memcached000 ns_1@172.23.105.169 07:39:41 - Mon Jul 13, 2015
      Bucket "UserInfo" loaded on node 'ns_1@172.23.105.168' in 0 seconds. ns_memcached000 ns_1@172.23.105.168 07:39:41 - Mon Jul 13, 2015
      Started rebalancing bucket UserInfo ns_rebalancer000 ns_1@172.23.105.167 07:39:40 - Mon Jul 13, 2015
      Starting rebalance, KeepNodes = ['ns_1@172.23.105.167','ns_1@172.23.105.170',
      'ns_1@172.23.105.168','ns_1@172.23.105.169'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator004 ns_1@172.23.105.167 07:39:38 - Mon Jul 13, 2015
      Node ns_1@172.23.105.169 joined cluster ns_cluster003 ns_1@172.23.105.169 07:36:45 - Mon Jul 13, 2015
      Couchbase Server has started on web port 8091 on node 'ns_1@172.23.105.169'. Version: "4.0.0-3402-enterprise". menelaus_sup001 ns_1@172.23.105.169 07:36:45 - Mon Jul 13, 2015
      Node 'ns_1@172.23.105.170' saw that node 'ns_1@172.23.105.169' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.170 07:36:37 - Mon Jul 13, 2015
      Node 'ns_1@172.23.105.168' saw that node 'ns_1@172.23.105.169' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.168 07:36:37 - Mon Jul 13, 2015
      Node 'ns_1@172.23.105.167' saw that node 'ns_1@172.23.105.169' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.167 07:36:36 - Mon Jul 13, 2015
      Started node add transaction by adding node 'ns_1@172.23.105.169' to nodes_wanted (group: 0) ns_cluster000 ns_1@172.23.105.167 07:36:34 - Mon Jul 13, 2015
      Node ns_1@172.23.105.168 joined cluster ns_cluster003 ns_1@172.23.105.168 07:36:24 - Mon Jul 13, 2015
      Couchbase Server has started on web port 8091 on node 'ns_1@172.23.105.168'. Version: "4.0.0-3402-enterprise". menelaus_sup001 ns_1@172.23.105.168 07:36:23 - Mon Jul 13, 2015
      Node 'ns_1@172.23.105.170' saw that node 'ns_1@172.23.105.168' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.170 07:36:14 - Mon Jul 13, 2015
      Node 'ns_1@172.23.105.167' saw that node 'ns_1@172.23.105.168' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.167 07:36:14 - Mon Jul 13, 2015
      Started node add transaction by adding node 'ns_1@172.23.105.168' to nodes_wanted (group: 0) ns_cluster000 ns_1@172.23.105.167 07:36:10 - Mon Jul 13, 2015
      Reset auto-failover count auto_failover000 ns_1@172.23.105.167 07:23:00 - Mon Jul 13, 2015
      Rebalance completed successfully.
      ns_orchestrator001 ns_1@172.23.105.167 07:22:59 - Mon Jul 13, 2015

      Attachments

        1. 172.23.105.169_mcstat.log
          24 kB
        2. cbcollect_info_172.23.105.167.zip
          42.44 MB
        3. MB-15652-1.png
          MB-15652-1.png
          97 kB
        4. MB-15652-2.png
          MB-15652-2.png
          78 kB
        5. MB-15652-3.png
          MB-15652-3.png
          68 kB
        6. MB-15652-4.png
          MB-15652-4.png
          71 kB
        7. Screen Shot 2015-07-20 at 11.29.40.png
          Screen Shot 2015-07-20 at 11.29.40.png
          143 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty