Details
Description
http://qa.hq.northscale.net/job/centos-32-2.0-swaprebalance-tests/336/consoleFull
2 tests failed:
./testrunner -i /tmp/swaprebalance-cent-32.ini GROUP=P0,get-cbcollect-info=True -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=1,num-buckets=4,num-swap=1,swap-orchestrator=True,GROUP=BASIC;P0
./testrunner -i /tmp/swaprebalance-cent-32.ini GROUP=P0,get-cbcollect-info=True -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=4,num-swap=1,swap-orchestrator=True,GROUP=TRUE;P0
Tue Mar 19 09:24:44.044616 PDT 3: (bucket-3) warmup completed in 7573 usec
2013-03-19 09:24:44.519 ns_memcached:4:info:message(ns_1@10.3.2.149) - Control connection to memcached on 'ns_1@10.3.2.149' disconnected: {{badmatch,
{error,
closed}},
[
,
,
,
,
,
]}
2013-03-19 09:24:44.525 ns_memcached:1:info:message(ns_1@10.3.2.148) - Bucket "bucket-3" loaded on node 'ns_1@10.3.2.148' in 0 seconds.
2013-03-19 09:24:44.527 ns_memcached:1:info:message(ns_1@10.3.2.147) - Bucket "bucket-3" loaded on node 'ns_1@10.3.2.147' in 0 seconds.
2013-03-19 09:24:45.030 ns_memcached:1:info:message(ns_1@10.3.2.149) - Bucket "bucket-3" loaded on node 'ns_1@10.3.2.149' in 0 seconds.
2013-03-19 09:24:48.196 ns_vbucket_mover:0:info:message(ns_1@10.3.2.145) - Bucket "bucket-3" rebalance appears to be swap rebalance
2013-03-19 09:24:59.379 mb_master:0:info:message(ns_1@10.3.2.146) - Haven't heard from a higher priority node or a master, so I'm taking over.
2013-03-19 09:25:10.343 ns_vbucket_mover:0:critical:message(ns_1@10.3.2.145) - <0.16483.9> exited with {badmatch,
{error,
{timeout,
{gen_server,call,
[timeout_diag_logger,
{diag,
{timeout,
2013-03-19 09:25:13.083 ns_orchestrator:2:info:message(ns_1@10.3.2.145) - Rebalance exited with reason {badmatch,
{error,
{timeout,
{gen_server,call,
[timeout_diag_logger,
{diag,
{timeout,
{gen_server,call,[ns_config,get]}
}}]}}}}
It seems that the crash occurred on 10.3.2.149, but there is no core files and logs from it:
[error_logger:error,2013-03-19T9:24:32.695,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Connection attempt from disallowed node 'ns_1@10.3.2.146' **
[error_logger:error,2013-03-19T9:24:32.698,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Connection attempt from disallowed node 'ns_1@10.3.2.152' **
[error_logger:error,2013-03-19T9:24:32.712,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Connection attempt from disallowed node 'ns_1@10.3.2.145' **
[error_logger:error,2013-03-19T9:24:32.716,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor:
Context: shutdown_error
Reason: normal
Offender: [{pid,<0.21125.0>},
{name,buckets_observing_subscription},
{mfargs,{ns_bucket_sup,subscribe_on_config_events,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]
[ns_server:error,2013-03-19T9:24:33.376,ns_1@10.3.2.149:ns_heart<0.25242.0>:ns_heart:grab_samples_loading_tasks:329]Failed to grab samples loader tasks: {exit,
{noproc,
{gen_server,call,
[samples_loader_tasks,get_tasks,
2000]}},
[{gen_server,call,3},
{ns_heart,grab_samples_loading_tasks,0},
{ns_heart,current_status,0},
{ns_heart,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
[ns_server:error,2013-03-19T9:24:33.404,ns_1@10.3.2.149:ns_heart<0.25242.0>:ns_heart:grab_samples_loading_tasks:329]Failed to grab samples loader tasks: {exit,
{noproc,
{gen_server,call,
[samples_loader_tasks,get_tasks,
2000]}},
[{gen_server,call,3},
{ns_heart,grab_samples_loading_tasks,0},
{ns_heart,current_status,0},
{ns_heart,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
[ns_server:error,2013-03-19T9:24:38.427,ns_1@10.3.2.149:<0.25384.0>:ns_orchestrator:rebalance_progress:163]Couldn't talk to orchestrator: {exit,
{timeout,
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
rebalance_progress,2000]}}}
[error_logger:error,2013-03-19T9:24:44.323,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.25302.0> terminating
** Last message in was {#Port<0.24508>,{exit_status,137}}
** When Server state == {state,#Port<0.24508>,memcached,
{["Tue Mar 19 09:24:44.044616 PDT 3: (bucket-3) warmup completed in 7573 usec", "Tue Mar 19 09:24:44.039001 PDT 3: (bucket-3) metadata loaded in 2070 usec", "Tue Mar 19 09:24:44.038500 PDT 3: (bucket-3) Failed to load mutation log, falling back to key dump", "Tue Mar 19 09:24:44.038254 PDT 3: Extension support isn't implemented in this version of bucket_engine", "Tue Mar 19 09:24:44.037866 PDT 3: (bucket-3) Warning: failed to load the engine session stats due to IO exception \"basic_ios::clear\"", "Tue Mar 19 09:24:44.032582 PDT 3: (bucket-3) Connected to mccouch: \"localhost:11213\"", "Tue Mar 19 09:24:44.031543 PDT 3: (bucket-3) Trying to connect to mccouch: \"localhost:11213\"", empty], [empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty,empty,empty,empty,empty,empty,empty, empty]},
undefined,[],0,true}
** Reason for termination ==
** {abnormal,137}
[error_logger:error,2013-03-19T9:24:44.330,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_port_server:init/1
pid: <0.25302.0>
registered_name: ns_port_memcached
exception exit: {abnormal,137}
in function gen_server:terminate/6
ancestors: [<0.25301.0>,ns_port_sup,ns_server_sup,ns_server_cluster_sup,
<0.59.0>]
messages: [{'EXIT',#Port<0.24508>,normal}]
links: [<0.25301.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 3523
neighbours:
[error_logger:error,2013-03-19T9:24:44.331,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.25301.0> terminating
** Last message in was {die,{abnormal,137}}
** When Server state == {state,memcached,5000,
{1363,710273,448093},
undefined,infinity}
** Reason for termination ==
** {abnormal,137}
[error_logger:error,2013-03-19T9:24:44.333,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: supervisor_cushion:init/1
pid: <0.25301.0>
registered_name: []
exception exit: {abnormal,137}
in function gen_server:terminate/6
ancestors: [ns_port_sup,ns_server_sup,ns_server_cluster_sup,<0.59.0>]
messages: []
links: [<0.25297.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 1591
neighbours:
[error_logger:error,2013-03-19T9:24:44.333,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_port_sup}
Context: child_terminated
Reason: {abnormal,137}
Offender: [{pid,<0.25301.0>},
{name,
{memcached,"/opt/couchbase/bin/memcached",
["-X",
"/opt/couchbase/lib/memcached/stdin_term_handler.so",
"-X",
"/opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/opt/couchbase/var/lib/couchbase/logs/memcached.log",
"-l","0.0.0.0:11210,0.0.0.0:11209:1000","-p",
"11210","-E",
"/opt/couchbase/lib/memcached/bucket_engine.so",
"-B","binary","-r","-c","10000","-e",
"admin=_admin;default_bucket_name=default;auto_create=false",
[]],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE", "/opt/couchbase/var/lib/couchbase/data/isasl.pw"},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,exit_status,
port_server_send_eol,stream]}},
{mfargs,
{erlang,apply,
[#Fun<ns_port_sup.3.119727222>,
[memcached,"/opt/couchbase/bin/memcached",
["-X",
"/opt/couchbase/lib/memcached/stdin_term_handler.so",
"-X",
"/opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/opt/couchbase/var/lib/couchbase/logs/memcached.log",
"-l","0.0.0.0:11210,0.0.0.0:11209:1000","-p",
"11210","-E",
"/opt/couchbase/lib/memcached/bucket_engine.so",
"-B","binary","-r","-c","10000","-e",
"admin=_admin;default_bucket_name=default;auto_create=false",
[]],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE", "/opt/couchbase/var/lib/couchbase/data/isasl.pw"},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,exit_status,
port_server_send_eol,stream]]]}},
{restart_type,permanent},
{shutdown,86400000},
{child_type,worker}]
[error_logger:error,2013-03-19T9:24:44.519,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server 'ns_memcached-bucket-3' terminating
** Last message in was check_started
** When Server state == {state,0,0,0,
{[],[]},
{[],[]},
{[],[]},
init,
{1363,710284,39926},
"bucket-3",#Port<0.24607>,
{interval,#Ref<0.0.2.44773>},
[{<0.25512.0>,#Ref<0.0.2.44930>},
{<0.25513.0>,#Ref<0.0.2.44922>},
{<0.25511.0>,#Ref<0.0.2.44921>},
{<0.25510.0>,#Ref<0.0.2.44919>}]}
** Reason for termination ==
** badmatch,{error,closed,
[{mc_client_binary,stats_recv,4},
{mc_client_binary,stats,4},
{ns_memcached,has_started,1},
{ns_memcached,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
[error_logger:error,2013-03-19T9:24:44.523,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_memcached:init/1
pid: <0.25508.0>
registered_name: 'ns_memcached-bucket-3'
exception exit: badmatch,{error,closed,
[{mc_client_binary,stats_recv,4},
{mc_client_binary,stats,4},
{ns_memcached,has_started,1},
{ns_memcached,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
in function gen_server:terminate/6
ancestors: ['single_bucket_sup-bucket-3',<0.25494.0>]
messages: []
links: [<0.25510.0>,<0.25512.0>,<0.25513.0>,<0.25511.0>,<0.95.0>,
<0.25495.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 317811
stack_size: 24
reductions: 26105
neighbours:
neighbour: [{pid,<0.25511.0>},
{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},
{ancestors,['ns_memcached-bucket-3', 'single_bucket_sup-bucket-3',<0.25494.0>]},
{messages,[]},
{links,[<0.25508.0>,#Port<0.24614>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,75025},
{stack_size,24},
{reductions,6949}]
neighbour: [{pid,<0.25513.0>},
{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},
{ancestors,['ns_memcached-bucket-3', 'single_bucket_sup-bucket-3',<0.25494.0>]},
{messages,[]},
{links,[<0.25508.0>,#Port<0.24610>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,75025},
{stack_size,24},
{reductions,6949}]
neighbour: [{pid,<0.25512.0>},
{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},
{ancestors,['ns_memcached-bucket-3', 'single_bucket_sup-bucket-3',<0.25494.0>]},
{messages,[]},
{links,[<0.25508.0>,#Port<0.24613>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,75025},
{stack_size,24},
{reductions,6949}]
neighbour: [{pid,<0.25510.0>},
{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},
{ancestors,['ns_memcached-bucket-3', 'single_bucket_sup-bucket-3',<0.25494.0>]},
{messages,[]},
{links,[<0.25508.0>,#Port<0.24611>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,75025},
{stack_size,24},
{reductions,6949}]
[error_logger:error,2013-03-19T9:24:44.524,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,'single_bucket_sup-bucket-3'}
Context: child_terminated
Reason: badmatch,{error,closed,
[{mc_client_binary,stats_recv,4},
{mc_client_binary,stats,4},
{ns_memcached,has_started,1},
{ns_memcached,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
Offender: [{pid,<0.25508.0>},
{name,{ns_memcached,"bucket-3"}},
{mfargs,{ns_memcached,start_link,["bucket-3"]}},
{restart_type,permanent},
{shutdown,86400000},
{child_type,worker}]
[ns_server:error,2013-03-19T9:24:56.352,ns_1@10.3.2.149:<0.25459.0>:ns_orchestrator:rebalance_progress:163]Couldn't talk to orchestrator: {exit,
{timeout,
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
rebalance_progress,2000]}}}
[ns_server:error,2013-03-19T9:25:00.363,ns_1@10.3.2.149:<0.25600.0>:ns_orchestrator:rebalance_progress:163]Couldn't talk to orchestrator: {exit,
{timeout,
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
rebalance_progress,2000]}}}
[error_logger:error,2013-03-19T9:25:47.527,ns_1@10.3.2.149:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_bucket_sup}
Context: shutdown_error
Reason: normal
Offender: [
,
,
{mfargs,{ns_bucket_sup,subscribe_on_config_events,[]}},
,
,
]