Details
Description
Tests to Reproduce:
./testrunner -i /tmp/nodes-win-64.ini get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete
Jenkins Ref Link:
http://qa.hq.northscale.net/job/win_2008_x64--01_00--qe-sanity-P0/16/consoleFull
Logs:
[user:info,2014-05-29T1:57:38.911,ns_1@10.3.2.239:<0.901.0>:ns_log:crash_consumption_loop:70]Port server xdcr_proxy on node 'babysitter_of_ns_1@127.0.0.1' exited with status 3. Restarting. Messages: [
{child_erlang,child_loop,2,[{file,"src/child_erlang.erl"},{line,121}]},
{child_erlang,child_start,1,[{file,"src/child_erlang.erl"},{line,78}]},
{init,start_it,1,[{file,"init.erl"},{line,1055}]},
{init,start_em,1,[{file,"init.erl"},{line,1034}]}]
[user:debug,2014-05-29T1:57:38.927,ns_1@10.3.2.239:<0.901.0>:ns_log:crash_consumption_loop:70]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: Failed to connect to 127.0.0.1 port 8091: Connection refused from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
WARNING: curl error: Failed to connect to 127.0.0.1 port 8091: Connection refused from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
EOL on stdin. Exiting
[ns_server:debug,2014-05-29T1:57:39.958,ns_1@10.3.2.239:<0.1090.0>:ns_pubsub:do_subscribe_link:136]Parent process of subscription {ns_config_events,<0.1088.0>} exited with reason {badmatch,
{is_pid,
false,
{badrpc,
{'EXIT',
{{badmatch,
{error,
{already_started, <13913.92.0>}}},
[{ns_child_ports_sup,
launch_port,
1,
[{file, "src/ns_child_ports_sup.erl"},
{line, 96}]},
{lists,
foreach,
2,
[{file, "lists.erl"},
{line, 1323}]},
{ns_child_ports_sup,
set_dynamic_children,
1,
[{file, "src/ns_child_ports_sup.erl"},
{line, 69}]},
{rpc,
'
5,
[{file, "rpc.erl"},
{line, 205}]}]}}}}}
[error_logger:error,2014-05-29T1:57:39.958,ns_1@10.3.2.239:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]
=========================CRASH REPORT=========================
crasher:
initial call: ns_ports_setup:setup_body_tramp/0
pid: <0.1088.0>
registered_name: ns_ports_setup
exception error: no match of right hand side value
{is_pid,false,
{badrpc,
{'EXIT',
{{badmatch,
{error,{already_started,<13913.92.0>}}},
[{ns_child_ports_sup,launch_port,1,
[{file,"src/ns_child_ports_sup.erl"},
{line,96}]},
{lists,foreach,2,
[{file,"lists.erl"},{line,1323}]},
{ns_child_ports_sup,set_dynamic_children,
1,
[{file,"src/ns_child_ports_sup.erl"},
{line,69}]},
{rpc,'
[{file,"rpc.erl"},{line,205}]}]}}}}
in function ns_ports_setup:set_childs_and_loop/1 (src/ns_ports_setup.erl, line 58)
in call from misc:delaying_crash/2 (src/misc.erl, line 1546)
ancestors: [ns_server_sup,ns_server_cluster_sup,<0.56.0>]
messages: [check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update]
links: [<0.896.0>,<0.1090.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 196650
stack_size: 27
reductions: 23593
neighbours:
[ns_server:debug,2014-05-29T1:57:54.958,ns_1@10.3.2.239:upr_consumer_conn-default-ns_1@10.3.3.38<0.1777.0>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.12548>
[error_logger:error,2014-05-29T1:57:54.958,ns_1@10.3.2.239:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]** Generic server <0.1777.0> terminating
** Last message in was {tcp,#Port<0.12548>, <<129,85,0,0,0,0,0,0,0,0,0,0,12,0,0,0,0,0,0,0,0,0, 0,0>>}
** When Server state == {state,#Port<0.12548>,
{consumer, "replication:ns_1@10.3.3.38->ns_1@10.3.2.239:default", 'ns_1@10.3.2.239',"default"},
<<>>,upr_consumer_conn,
{state,idle,[]},
#Port<0.12550>,<0.1779.0>}
** Reason for termination ==
** badmatch,{error,closed,
[{upr_proxy,handle_packet,2,[{file,"src/upr_proxy.erl"},{line,130}]},
{mc_replication,process_data,3,
[{file,"src/mc_replication.erl"},{line,59}]},
{mc_replication,process_data,4,
[{file,"src/mc_replication.erl"},{line,76}]},
{upr_proxy,handle_info,2,[{file,"src/upr_proxy.erl"},{line,84}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,604}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
[error_logger:error,2014-05-29T1:57:54.958,ns_1@10.3.2.239:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]
=========================CRASH REPORT=========================
crasher:
initial call: upr_proxy:init/1
pid: <0.1777.0>
registered_name: 'upr_consumer_conn-default-ns_1@10.3.3.38'
exception exit: badmatch,{error,closed,
[{upr_proxy,handle_packet,2,
[{file,"src/upr_proxy.erl"},{line,130}]},
{mc_replication,process_data,3,
[{file,"src/mc_replication.erl"},
{line,59}]},
{mc_replication,process_data,4,
[{file,"src/mc_replication.erl"},
{line,76}]},
{upr_proxy,handle_info,2,
[{file,"src/upr_proxy.erl"},{line,84}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,604}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: ['upr_replicator-default-ns_1@10.3.3.38','upr_sup-default',
'single_bucket_sup-default',<0.1150.0>]
messages: [{'EXIT',<0.1776.0>,shutdown}]
links: [<0.1776.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 2586
stack_size: 27
reductions: 243693
neighbours:
[user:debug,2014-05-29T1:59:44.240,ns_1@10.3.2.239:<0.901.0>:ns_log:crash_consumption_loop:70]Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: Thu May 29 01:59:43.126953 Pacific Daylight Time 3: (default) UPR (Producer) eq_uprq:replication:ns_1@10.3.2.239->ns_1@10.3.2.243:default - (vb 84) Stream closing, 0 items sent from disk, 0 items sent from memory, 55 was last seqno sent
Thu May 29 01:59:43.126953 Pacific Daylight Time 3: (default) UPR (Producer) eq_uprq:replication:ns_1@10.3.2.239->ns_1@10.3.2.243:default - (vb 85) Stream closing, 0 items sent from disk, 0 items sent from memory, 56 was last seqno sent
Thu May 29 01:59:43.155273 Pacific Daylight Time 3: (default) Shutting down tap connections!
Thu May 29 01:59:43.155273 Pacific Daylight Time 3: (default) Shutting down upr connections!
'die!' on stdin. Exiting super-quickly
[user:info,2014-05-29T2:00:00.880,ns_1@10.3.2.239:<0.9767.0>:ns_log:crash_consumption_loop:70]Port server xdcr_proxy on node 'babysitter_of_ns_1@127.0.0.1' exited with status 3. Restarting. Messages: [{ns_ssl_proxy,stop,[],[]}
,
{child_erlang,child_loop,2,[
{child_erlang,child_start,1,[{file,"src/child_erlang.erl"}
,
{line,78}]},
{init,start_it,1,[
{init,start_em,1,[{file,"init.erl"}
,
{line,1034}]}]
[user:debug,2014-05-29T2:00:00.896,ns_1@10.3.2.239:<0.9767.0>:ns_log:crash_consumption_loop:70]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
WARNING: curl error: Failed to connect to 127.0.0.1 port 8091: Connection refused from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
WARNING: curl error: Failed to connect to 127.0.0.1 port 8091: Connection refused from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
EOL on stdin. Exiting
[ns_server:debug,2014-05-29T2:00:01.458,ns_1@10.3.2.239:ns_heart_slow_status_updater<0.9801.0>:ns_heart:current_status_slow_inner:259]Ignoring failure to grab system stats:
Uploading Logs.