Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15699

[system tests] Rebalance out 7-1 nodes exited with reason {unexpected_exit after Service 'memcached' exited with status 134

    XMLWordPrintable

Details

    • Untriaged
    • Ubuntu 64-bit
    • Unknown
    • Mar 9 - Mar 27

    Description

      system tests with 2 buckets * 2 views

      rebalance 7-1 node:

      Shutting down bucket "saslbucket" on 'ns_1@172.23.105.22' for deletion ns_memcached000 ns_1@172.23.105.22 07:19:15 - Wed Jul 15, 2015
      Bucket "saslbucket" loaded on node 'ns_1@172.23.105.157' in 134 seconds. ns_memcached000 ns_1@172.23.105.157 07:18:07 - Wed Jul 15, 2015
      Bucket "default" loaded on node 'ns_1@172.23.105.157' in 88 seconds. ns_memcached000 ns_1@172.23.105.157 07:17:22 - Wed Jul 15, 2015
      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.2752.10>,
      badmatch,{error,closed,
      {gen_server,call,
      [

      {'janitor_agent-default', 'ns_1@172.23.105.157'}

      ,
      {if_rebalance,<0.26733.9>,
      {dcp_takeover,'ns_1@172.23.105.22',504}},
      infinity]}}}}
      ns_orchestrator002 ns_1@172.23.105.156 07:16:28 - Wed Jul 15, 2015
      <0.2150.10> exited with {unexpected_exit,
      {'EXIT',<0.2752.10>,
      badmatch,{error,closed,
      {gen_server,call,
      [

      {'janitor_agent-default','ns_1@172.23.105.157'}

      ,
      {if_rebalance,<0.26733.9>,
      {dcp_takeover,'ns_1@172.23.105.22',504}},
      infinity]}}}} ns_vbucket_mover000 ns_1@172.23.105.156 07:16:28 - Wed Jul 15, 2015
      Control connection to memcached on 'ns_1@172.23.105.157' disconnected: {{badmatch,
      {error,
      closed}},
      [{mc_client_binary,
      cmd_vocal_recv,
      5,
      [

      {file, "src/mc_client_binary.erl"}, {line, 156}]},
      {mc_client_binary,
      select_bucket,
      2,
      [{file, "src/mc_client_binary.erl"}

      ,

      {line, 351}]},
      {ns_memcached,
      ensure_bucket,
      2,
      [{file, "src/ns_memcached.erl"}, {line, 1292}]},
      {ns_memcached,
      handle_info,
      2,
      [{file, "src/ns_memcached.erl"}, {line, 745}]},
      {gen_server,
      handle_msg,
      5,
      [{file, "gen_server.erl"}, {line, 604}]},
      {ns_memcached,
      init,
      1,
      [{file, "src/ns_memcached.erl"}, {line, 174}]},
      {gen_server,
      init_it,
      6,
      [{file, "gen_server.erl"}, {line, 304}]},
      {proc_lib,
      init_p_do_apply,
      3,
      [{file, "proc_lib.erl"}, {line, 239}]}]} (repeated 1 times) ns_memcached000 ns_1@172.23.105.157 07:16:09 - Wed Jul 15, 2015
      Service 'memcached' exited with status 134. Restarting. Messages: 2015-07-15T07:15:33.412636-07:00 WARNING /opt/couchbase/lib/memcached/ep.so() [0x7f3905f4e2a7]
      2015-07-15T07:15:33.412664-07:00 WARNING /opt/couchbase/lib/memcached/ep.so() [0x7f3905f4e8f9]
      2015-07-15T07:15:33.412690-07:00 WARNING /opt/couchbase/bin/../lib/libplatform.so.0.1.0() [0x7f39115bbbb8]
      2015-07-15T07:15:33.412723-07:00 WARNING /lib64/libpthread.so.0() [0x7f390fd26851]
      2015-07-15T07:15:33.412782-07:00 WARNING /lib64/libc.so.6(clone+0x6d) [0x7f390eec890d] ns_log000 ns_1@172.23.105.157 07:15:50 - Wed Jul 15, 2015
      Control connection to memcached on 'ns_1@172.23.105.157' disconnected: {{badmatch,
      {error,
      closed}},
      [{mc_client_binary,
      cmd_vocal_recv,
      5,
      [{file, "src/mc_client_binary.erl"}, {line, 156}]},
      {mc_client_binary,
      select_bucket,
      2,
      [{file, "src/mc_client_binary.erl"},{line, 351}

      ]},
      {ns_memcached,
      ensure_bucket,
      2,
      [

      {file, "src/ns_memcached.erl"}, {line, 1292}]},
      {ns_memcached,
      handle_info,
      2,
      [{file, "src/ns_memcached.erl"}

      ,

      {line, 745}

      ]},
      {gen_server,
      handle_msg,
      5,
      [

      {file, "gen_server.erl"}, {line, 604}]},
      {ns_memcached,
      init,
      1,
      [{file, "src/ns_memcached.erl"}, {line, 174}]},
      {gen_server,
      init_it,
      6,
      [{file, "gen_server.erl"}

      ,

      {line, 304}

      ]},
      {proc_lib,
      init_p_do_apply,
      3,
      [

      {file, "proc_lib.erl"}

      ,

      {line, 239}

      ]}]} ns_memcached000 ns_1@172.23.105.157 07:15:50 - Wed Jul 15, 2015
      Bucket "default" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.156 07:14:31 - Wed Jul 15, 2015
      Started rebalancing bucket default ns_rebalancer000 ns_1@172.23.105.156 07:14:27 - Wed Jul 15, 2015
      Node 'ns_1@172.23.105.207' saw that node 'ns_1@172.23.105.159' went down. Details: [

      {nodedown_reason, connection_closed}] ns_node_disco005 ns_1@172.23.105.207 07:06:44 - Wed Jul 15, 2015
      Node 'ns_1@172.23.105.22' saw that node 'ns_1@172.23.105.159' went down. Details: [{nodedown_reason, connection_closed}

      ] ns_node_disco005 ns_1@172.23.105.22 07:05:33 - Wed Jul 15, 2015
      Bucket "saslbucket" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.156 07:01:32 - Wed Jul 15, 2015
      Started rebalancing bucket saslbucket ns_rebalancer000 ns_1@172.23.105.156 07:01:32 - Wed Jul 15, 2015
      Starting rebalance, KeepNodes = ['ns_1@172.23.105.160','ns_1@172.23.105.156',
      'ns_1@172.23.105.206','ns_1@172.23.105.158',
      'ns_1@172.23.105.157','ns_1@172.23.105.207'], EjectNodes = ['ns_1@172.23.105.22'], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator004 ns_1@172.23.105.156 07:01:32 - Wed Jul 15

      will provide logs soon

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty