[ns_server:debug,2014-07-22T23:50:06.491,ns_1@172.23.106.201:upr_consumer_conn-default-ns_1@172.23.106.197<0.1380.0>:upr_proxy:handle_info:88]Socket #Port<0.7944> was closed. Closing myself. State = {state, #Port<0.7944>, {consumer, "replication:ns_1@172.23.106.197->ns_1@172.23.106.201:default", 'ns_1@172.23.106.201', "default"}, <<>>, upr_consumer_conn, {state,idle, [0,1,2,3,4,5,6,7,8, 9,10,11,12,13,14, 15,16,17,18,19,20, 21,22,23,24,25,26, 27,28,29,30,31,32, 33,34,35,36,37,38, 39,40,41,42,43,44, 45,46,47,48,49,50, 51,52,53,54,55,56, 57,58,59,60,61,62, 63,64,65,66,67,68, 69,70,71,72,73,74, 75,76,77,78,79,80, 81,82,83,84,85,86, 87,88,89,90,91,92, 93,94,95,96,97,98, 99,100,101,102, 103,104,105,106, 107,108,109,110, 111,112,113,114, 115,116,117,118, 119,120,121,122, 123,124,125,126, 127,128,129,130, 131,132,133,134, 135,136,137,138, 139,140,141,142, 143,144,145,146, 147,148,149,150, 151,152,153,154, 155,156,157,158, 159,160,161,162, 163,164,165,166, 167,168,169,170, 171,172,173,174, 175,176,177,178, 179,180,181,182, 183,184,185,186, 187,188,189,190, 191,192,193,194, 195,196,197,198, 199,200,201,202, 203,204,205,206, 207,208,209,210, 211,212,213,214, 215,216,217,218, 219,220,221,222, 223,224,225,226, 227,228,229,230, 231,232,233,234, 235,236,237,238, 239,240,241,242, 243,244,245,246, 247,248,249,250, 251,252,253,254, 255]}, #Port<0.7945>, <0.1390.0>} [ns_server:debug,2014-07-22T23:50:06.492,ns_1@172.23.106.201:upr_consumer_conn-default-ns_1@172.23.106.197<0.1380.0>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.7944> [ns_server:debug,2014-07-22T23:50:06.493,ns_1@172.23.106.201:<0.1390.0>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.7945> [user:info,2014-07-22T23:50:06.497,ns_1@172.23.106.201:<0.846.0>:ns_log:crash_consumption_loop:70]Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 137. Restarting. Messages: Tue Jul 22 23:49:43.534507 PDT 3: (default) UPR (Consumer) eq_uprq:replication:ns_1@172.23.106.197->ns_1@172.23.106.201:default - (vb 434) Attempting to add takeover stream with start seqno 7152, end seqno 18446744073709551615, vbucket uuid 157228383951054, snap start seqno 7152, and snap end seqno 7152 Tue Jul 22 23:49:43.544875 PDT 3: (default) UPR (Consumer) eq_uprq:replication:ns_1@172.23.106.197->ns_1@172.23.106.201:default - (vb 419) Attempting to add takeover stream with start seqno 7455, end seqno 18446744073709551615, vbucket uuid 107021536065092, snap start seqno 7455, and snap end seqno 7455 Tue Jul 22 23:49:43.554326 PDT 3: (default) UPR (Consumer) eq_uprq:replication:ns_1@172.23.106.197->ns_1@172.23.106.201:default - (vb 418) Attempting to add takeover stream with start seqno 7361, end seqno 18446744073709551615, vbucket uuid 102391713229103, snap start seqno 7361, and snap end seqno 7361 Tue Jul 22 23:49:43.567990 PDT 3: (default) UPR (Consumer) eq_uprq:replication:ns_1@172.23.106.197->ns_1@172.23.106.201:default - (vb 417) Attempting to add takeover stream with start seqno 7198, end seqno 18446744073709551615, vbucket uuid 122231717609681, snap start seqno 7198, and snap end seqno 7198 Tue Jul 22 23:49:43.579715 PDT 3: (default) UPR (Consumer) eq_uprq:replication:ns_1@172.23.106.197->ns_1@172.23.106.201:default - (vb 435) Attempting to add takeover stream with start seqno 7199, end seqno 18446744073709551615, vbucket uuid 179024591545779, snap start seqno 7199, and snap end seqno 7199 [ns_server:debug,2014-07-22T23:50:07.070,ns_1@172.23.106.201:ns_memcached-default<0.667.0>:ns_memcached:handle_info:754]Got {'EXIT',<0.695.0>,{badmatch,{error,closed}}}. Exiting. [user:info,2014-07-22T23:50:07.071,ns_1@172.23.106.201:ns_memcached-default<0.667.0>:ns_memcached:terminate:815]Control connection to memcached on 'ns_1@172.23.106.201' disconnected: {badmatch, {error, closed}} [error_logger:error,2014-07-22T23:50:07.071,ns_1@172.23.106.201:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: erlang:apply/2 pid: <0.695.0> registered_name: [] exception error: no match of right hand side value {error,closed} in function mc_binary:quick_stats_recv/3 (src/mc_binary.erl, line 56) in call from mc_binary:quick_stats_loop/5 (src/mc_binary.erl, line 156) in call from mc_binary:quick_stats/5 (src/mc_binary.erl, line 141) in call from ns_memcached:do_handle_call/3 (src/ns_memcached.erl, line 576) in call from ns_memcached:worker_loop/3 (src/ns_memcached.erl, line 209) ancestors: ['ns_memcached-default','single_bucket_sup-default', <0.653.0>] messages: [] links: [<0.667.0>] dictionary: [] trap_exit: false status: running heap_size: 6772 stack_size: 27 reductions: 136960976 neighbours: [error_logger:error,2014-07-22T23:50:07.072,ns_1@172.23.106.201:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.667.0> terminating ** Last message in was {'EXIT',<0.695.0>,{badmatch,{error,closed}}} ** When Server state == {state,1,0,0, {[],[]}, {[],[]}, {[],[]}, warmed, {1406,47439,796333}, "default",#Port<0.7423>, {interval,#Ref<0.0.0.5719>}, [{<0.696.0>,#Ref<0.0.6025.134176>}, {<0.693.0>,#Ref<0.0.5854.35640>}, {<0.694.0>,#Ref<0.0.982.213423>}], []} ** Reason for termination == ** {badmatch,{error,closed}} [stats:error,2014-07-22T23:50:07.072,ns_1@172.23.106.201:<0.680.0>:stats_collector:handle_info:124]Exception in stats collector: {exit, {{badmatch,{error,closed}}, {gen_server,call, ['ns_memcached-default', {stats,<<>>}, 180000]}}, [{gen_server,call,3, [{file,"gen_server.erl"},{line,188}]}, {ns_memcached,do_call,3, [{file,"src/ns_memcached.erl"}, {line,1401}]}, {stats_collector,grab_all_stats,1, [{file,"src/stats_collector.erl"}, {line,84}]}, {stats_collector,handle_info,2, [{file,"src/stats_collector.erl"}, {line,116}]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,604}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]} [ns_server:debug,2014-07-22T23:50:07.072,ns_1@172.23.106.201:ns_memcached-default<0.10055.16>:ns_memcached:init:161]Starting ns_memcached [ns_server:debug,2014-07-22T23:50:07.073,ns_1@172.23.106.201:<0.9940.16>:ns_memcached:run_connect_phase:184]Started 'connecting' phase of ns_memcached-default. Parent is <0.10055.16> [error_logger:error,2014-07-22T23:50:07.073,ns_1@172.23.106.201:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: ns_memcached:init/1 pid: <0.667.0> registered_name: [] exception exit: {badmatch,{error,closed}} in function gen_server:init_it/6 (gen_server.erl, line 328) ancestors: ['single_bucket_sup-default',<0.653.0>] messages: [] links: [<0.693.0>,<0.694.0>,<0.696.0>,<0.277.0>,<0.654.0>] dictionary: [] trap_exit: true status: running heap_size: 75113 stack_size: 27 reductions: 62719019 neighbours: neighbour: [{pid,<0.696.0>}, {registered_name,[]}, {initial_call,{erlang,apply,['Argument__1','Argument__2']}}, {current_function,{gen,do_call,4}}, {ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.653.0>]}, {messages,[]}, {links,[<0.667.0>,#Port<0.7518>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,6772}, {stack_size,23}, {reductions,117288907}] neighbour: [{pid,<0.694.0>}, {registered_name,[]}, {initial_call,{erlang,apply,['Argument__1','Argument__2']}}, {current_function,{gen,do_call,4}}, {ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.653.0>]}, {messages,[]}, {links,[<0.667.0>,#Port<0.7519>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,4185}, {stack_size,23}, {reductions,55876512}] neighbour: [{pid,<0.693.0>}, {registered_name,[]}, {initial_call,{erlang,apply,['Argument__1','Argument__2']}}, {current_function,{gen,do_call,4}}, {ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.653.0>]}, {messages,[]}, {links,[<0.667.0>,#Port<0.7521>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,233}, {stack_size,23}, {reductions,176160903}] [error_logger:error,2014-07-22T23:50:07.074,ns_1@172.23.106.201:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]Supervisor received unexpected message: {ack,<0.667.0>, {error,{badmatch,{error,closed}}}} [error_logger:error,2014-07-22T23:50:10.455,ns_1@172.23.106.201:error_logger<0.6.0>:ale_error_logger_handler:do_log:203] =========================CRASH REPORT========================= crasher: initial call: erlang:apply/2 pid: <0.30306.15> registered_name: [] exception exit: stopped in function ns_rebalancer:run_mover/7 (src/ns_rebalancer.erl, line 514) in call from ns_rebalancer:rebalance/6 (src/ns_rebalancer.erl, line 478) in call from ns_rebalancer:'-rebalance/5-fun-1-'/6 (src/ns_rebalancer.erl, line 436) in call from lists:foreach/2 (lists.erl, line 1323) in call from ns_rebalancer:rebalance/5 (src/ns_rebalancer.erl, line 388) ancestors: [<0.814.0>,mb_master_sup,mb_master,ns_server_sup, ns_server_cluster_sup,<0.60.0>] messages: [] links: [<0.30321.15>,<0.814.0>] dictionary: [{random_seed,{8236,26623,17360}}] trap_exit: false status: running heap_size: 121536 stack_size: 27 reductions: 3307285 neighbours: neighbour: [{pid,<0.30323.15>}, {registered_name,[]}, {initial_call,{erlang,apply,['Argument__1','Argument__2']}}, {current_function, {ns_rebalance_observer,docs_left_updater_loop,1}}, {ancestors, [<0.30321.15>,<0.30306.15>,<0.814.0>,mb_master_sup, mb_master,ns_server_sup,ns_server_cluster_sup, <0.60.0>]}, {messages,[]}, {links,[<0.30321.15>,<0.277.0>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,75113}, {stack_size,6}, {reductions,469790}] neighbour: [{pid,<0.30321.15>}, {registered_name,[]}, {initial_call,{ns_rebalance_observer,init,['Argument__1']}}, {current_function,{gen_server,loop,6}}, {ancestors,[<0.30306.15>,<0.814.0>,mb_master_sup,mb_master, ns_server_sup,ns_server_cluster_sup,<0.60.0>]}, {messages,[]}, {links,[<0.30322.15>,<0.30323.15>,<0.30306.15>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,75113}, {stack_size,9}, {reductions,1515105}]