Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15926

[ec2] memcached crash forced because of race scenario induced by cluster manager

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 3.1.1, 4.0.0
    • 4.0.0
    • ns_server
    • Security Level: Public
    • centos 6.5 64-bit, 8 core cpu, 30 GB RAM, 80 GB SSD, 10 Git network

    Description

      Install cb server 4.0.0-3570 on 7 nodes
      Create cluster of 7 nodees (5 kv, 1 index and 1 n1ql)
      Load 34 million items with size 1K to default bucket (no view nor indexing)
      Install cb server 4.0.0-3570 on other 7 nodes with same services setup above (5,1,1)
      Addd new 7 nodes to cluster. Rebalance failed in few minutes.
      Check log see memcached crashed.

      <0.27899.12> exited with {unexpected_exit,
      {'EXIT',<0.27907.12>,
      {bulk_set_vbucket_state_failed,
      [{'ns_1@10.145.15.54',
      {'EXIT',
      {{{{case_clause,
      {error,
      {{{badmatch,
      {error,
      badmatch,{error,etimedout,
      [{mc_replication,connect,1,
      [

      {file,"src/mc_replication.erl"}, {line,30}]},
      {mc_replication,connect,1,
      [{file,"src/mc_replication.erl"}

      ,

      {line,49}]},
      {dcp_proxy,connect,4,
      [{file,"src/dcp_proxy.erl"}, {line,174}]},
      {dcp_proxy,maybe_connect,1,
      [{file,"src/dcp_proxy.erl"}, {line,161}]},
      {dcp_producer_conn,init,2,
      [{file,"src/dcp_producer_conn.erl"}, {line,30}]},
      {dcp_proxy,init,1,
      [{file,"src/dcp_proxy.erl"}, {line,46}]},
      {gen_server,init_it,6,
      [{file,"gen_server.erl"}, {line,304}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"}, {line,239}]}]}}},
      [{dcp_replicator,init,1,
      [{file,"src/dcp_replicator.erl"},{line,49}

      ]},
      {gen_server,init_it,6,
      [

      {file,"gen_server.erl"},{line,304}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,239}]}]},
      {child,undefined,'ns_1@10.164.216.81', {dcp_replicator,start_link, ['ns_1@10.164.216.81',"default"]},
      temporary,60000,worker,
      [dcp_replicator]}}}},
      [{dcp_sup,start_replicator,2,
      [{file,"src/dcp_sup.erl"},{line,53}]},
      {dcp_sup,
      'manage_replicators/2-lc$^2/1-2',2,
      [{file,"src/dcp_sup.erl"},{line,69}]      },
      {dcp_replication_manager,handle_call,3,
      [{file,"src/dcp_replication_manager.erl"}, {line,78}]},
      {gen_server,handle_msg,5,
      [{file,"gen_server.erl"}

      ,

      {line,585}

      ]},
      {proc_lib,init_p_do_apply,3,
      [

      {file,"proc_lib.erl"}

      ,

      {line,239}

      ]}]},
      {gen_server,call,
      ['dcp_replication_manager-default',

      {get_replicator_pid,408}

      ,
      infinity]}},
      {gen_server,call,
      [

      {'janitor_agent-default', 'ns_1@10.145.15.54'}

      ,
      {if_rebalance,<0.9128.11>,
      {update_vbucket_state,458,replica,passive,
      'ns_1@10.165.120.25'}},
      infinity]}}}}]}}}
      ns_vbucket_mover000
      ns_1@10.233.98.253
      09:28:38 - Fri Jul 31, 2015
      Node 'ns_1@10.233.170.117' saw that node 'ns_1@10.164.216.81' came up. Tags: []
      ns_node_disco004
      ns_1@10.233.170.117
      09:02:44 - Fri Jul 31, 2015
      Node 'ns_1@10.164.216.81' saw that node 'ns_1@10.233.170.117' came up. Tags: []
      ns_node_disco004
      ns_1@10.164.216.81
      09:02:44 - Fri Jul 31, 2015
      Bucket "default" rebalance does not seem to be swap rebalance
      ns_vbucket_mover000
      ns_1@10.233.98.253
      09:01:38 - Fri Jul 31, 2015

      Attachments

        Issue Links

          blocks

          Task - A task that needs to be done. MB-15886 3.1.1 Minor Release

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhinav Abhi Dangeti
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty