memcached process does not restart successfully after configuration change

In implementing the UI for memory quota, a parameter for max_size was added. It adds the system memory by default at startup, but during the wizard if the user changes it from "unlimited" to a smaller value, the port is terminated and restarted with the new param.

The process exits right after restart.

Below are the relevant log entries from the moment right after the wizard screen sets the memory quota to 100MiByte from the default.

The code for setting memory is in the following change:
http://review.northscale.com:8080/#change,749

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
unsupervising port: {memcached,"./bin/memcached/memcached",
["-p","11210","-X",
"./bin/memcached/stdin_term_handler.so","-E",
"./bin/ep_engine/ep.so","-B","binary","-r",
"-e",
"vb0=false;ht_size=12289;ht_locks=23;dbname=/Users/ingenthr/src/ns_server/data/ns_1/default;max_size=3141497600"],
[{env,[

{"MEMCACHED_TOP_KEYS","100"}]},
use_stdio,stderr_to_stdout,stream]}

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_port_init terminating: {error,
{'EXIT',
{undef,
[{ns_port_sup,launch_port,
[memcached,"./bin/memcached/memcached",
["-p","11210","-X",
"./bin/memcached/stdin_term_handler.so","-E",
"./bin/ep_engine/ep.so","-B","binary","-r",
"-e",
"vb0=false;ht_size=12289;ht_locks=23;dbname=/Users/ingenthr/src/ns_server/data/ns_1/default;max_size=104857600"],
[{env,[{"MEMCACHED_TOP_KEYS","100"}

]},
use_stdio,stderr_to_stdout,stream]]},

{lists,foreach,2},
{ns_port_init,reconfig,1},
{ns_port_init,handle_event,2},
{gen_event,server_update,4},
{gen_event,server_notify,4},
{gen_event,server_notify,4},
{gen_event,handle_msg,5}]}}}...

=ERROR REPORT==== 20-Jun-2010::13:26:40 ===
** gen_event handler ns_port_init crashed.
** Was installed in ns_config_events
** Last event was: {{node,'ns_1@127.0.0.1',memcached},
[{port,11210},
{ht_size,12289},
{ht_locks,23},
{dbname,"/Users/ingenthr/src/ns_server/data/ns_1/default"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{max_size,100}]}
** When handler state == {state}
** Reason == {'function not exported',
[{ns_port_sup,launch_port,
[memcached,"./bin/memcached/memcached",
["-p","11210","-X",
"./bin/memcached/stdin_term_handler.so","-E",
"./bin/ep_engine/ep.so","-B","binary","-r","-e",
"vb0=false;ht_size=12289;ht_locks=23;dbname=/Users/ingenthr/src/ns_server/data/ns_1/default;max_size=104857600"],
[{env,[{"MEMCACHED_TOP_KEYS","100"}]},
use_stdio,stderr_to_stdout,stream]]},
{lists,foreach,2}

,

{ns_port_init,reconfig,1}

,

{ns_port_init,handle_event,2}

,

{gen_event,server_update,4}

,

{gen_event,server_notify,4},
{gen_event,server_notify,4}

,

{gen_event,handle_msg,5}

]}

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
config change:

{node,'ns_1@127.0.0.1',memcached}

-> [

{port,11210}

,

{ht_size,12289}

,

{ht_locks,23}

,

{dbname,[47,85,115,101,114,115,47,105,110,103,101,110,116,104,114,47,115,114,99,47,110,115,95,115,101,114,118,101,114,47,100,97,116,97,47,110,115,95,49,47,100,101,102,97,117,108,116]}

,

{admin_user,[95,97,100,109,105,110]}

,

{admin_pass,[95,97,100,109,105,110]}

,

{max_size,100}

]

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_node_disco_conf_events config all

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
Pushing config

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_log: logging menelaus_web:104:initStatus updated to "cluster".

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
Pushing config done

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_log: logging ns_config_log:1:Config changed on node 'ns_1@127.0.0.1'

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_log: suppressing duplicate log ns_config_log:1("Config changed on node 'ns_1@127.0.0.1'") because it's been seen 10 times in the past 27.16596 secs (last seen 18.489197 secs ago

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
config change: init_status -> [

{value,[99,108,117,115,116,101,114]}

]

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_node_disco_conf_events config all

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
Pushing config

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
Pushing config done

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_log: logging ns_config_log:1:Config changed on node 'ns_1@127.0.0.1'

=INFO REPORT==== 20-Jun-2010::13:26:40 ===
ns_log: suppressing duplicate log ns_config_log:1("Config changed on node 'ns_1@127.0.0.1'") because it's been seen 11 times in the past 27.167144 secs (last seen 0.001184 secs ago

=ERROR REPORT==== 20-Jun-2010::13:26:45 ===
ns_memcached:terminate(badmatch,{error,closed,
[

{mc_client_binary,stats_recv,3},
{mc_client_binary,stats,2},
{ns_memcached,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}, {state,#Port<0.2291>})

=ERROR REPORT==== 20-Jun-2010::13:26:47 ===
** Generic server ns_memcached terminating
** Last message in was {stats,"default","vbucket"}
** When Server state == {state,#Port<0.2291>}
** Reason for termination ==
** badmatch,{error,closed,
[{mc_client_binary,stats_recv,3}

,

{mc_client_binary,stats,2}

,

{ns_memcached,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}

=INFO REPORT==== 20-Jun-2010::13:26:47 ===
ns_janitor:current_states: can't reach nodes ['ns_1@127.0.0.1']: trying again

=INFO REPORT==== 20-Jun-2010::13:26:48 ===
ns_memcached connecting to memcached on port 11210

=ERROR REPORT==== 20-Jun-2010::13:26:48 ===
ns_memcached:terminate(badmatch,{error,econnrefused,
[{ns_memcached,connect,1},
{ns_memcached,handle_call,3}

,

{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}, {state,undefined})

=ERROR REPORT==== 20-Jun-2010::13:26:50 ===
** Generic server ns_memcached terminating
** Last message in was {stats,"default","vbucket"}
** When Server state == {state,undefined}
** Reason for termination ==
** badmatch,{error,econnrefused,
[{ns_memcached,connect,1},
{ns_memcached,handle_call,3},
{gen_server,handle_msg,5}

,

{proc_lib,init_p_do_apply,3}]}

=INFO REPORT==== 20-Jun-2010::13:26:50 ===
ns_janitor:current_states: can't reach nodes ['ns_1@127.0.0.1']: trying again

=INFO REPORT==== 20-Jun-2010::13:26:51 ===
ns_memcached connecting to memcached on port 11210

=ERROR REPORT==== 20-Jun-2010::13:26:51 ===
ns_memcached:terminate(badmatch,{error,econnrefused,
[{ns_memcached,connect,1},
{ns_memcached,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}

]},

{state,undefined})

=ERROR REPORT==== 20-Jun-2010::13:26:53 ===
** Generic server ns_memcached terminating
** Last message in was {stats,"default","vbucket"}
** When Server state == {state,undefined}

• • Reason for termination ==
  • badmatch,{error,econnrefused,
    [ {ns_memcached,connect,1},
    {ns_memcached,handle_call,3},
    {gen_server,handle_msg,5},
    {proc_lib,init_p_do_apply,3}]}
    
    =INFO REPORT==== 20-Jun-2010::13:26:53 ===
    ns_janitor:current_states: can't reach nodes ['ns_1@127.0.0.1']: trying again
    
    =INFO REPORT==== 20-Jun-2010::13:26:54 ===
    ns_memcached connecting to memcached on port 11210
    
    =ERROR REPORT==== 20-Jun-2010::13:26:54 ===
    ns_memcached:terminate(badmatch,{error,econnrefused,
    [{ns_memcached,connect,1}

    ,
    

    {ns_memcached,handle_call,3},
    {gen_server,handle_msg,5},
    {proc_lib,init_p_do_apply,3}]}, {state,undefined})
    
    =ERROR REPORT==== 20-Jun-2010::13:26:56 ===
    ** Generic server ns_memcached terminating
    ** Last message in was {stats,"default","vbucket"}
    ** When Server state == {state,undefined}
    ** Reason for termination ==
    ** badmatch,{error,econnrefused,
    [{ns_memcached,connect,1},
    {ns_memcached,handle_call,3}

    ,
    

    {gen_server,handle_msg,5},
    {proc_lib,init_p_do_apply,3}]}
    
    =INFO REPORT==== 20-Jun-2010::13:26:56 ===
    ns_janitor:current_states: can't reach nodes ['ns_1@127.0.0.1']: trying again
    
    =INFO REPORT==== 20-Jun-2010::13:26:57 ===
    ns_memcached connecting to memcached on port 11210
    
    =ERROR REPORT==== 20-Jun-2010::13:26:57 ===
    ns_memcached:terminate(badmatch,{error,econnrefused,
    [{ns_memcached,connect,1},
    {ns_memcached,handle_call,3},
    {gen_server,handle_msg,5}

    ,
    

    {proc_lib,init_p_do_apply,3}]}, {state,undefined})
    
    =ERROR REPORT==== 20-Jun-2010::13:26:59 ===
    ** Generic server ns_memcached terminating
    ** Last message in was {stats,"default","vbucket"}
    ** When Server state == {state,undefined}
    ** Reason for termination ==
    ** badmatch,{error,econnrefused,
    [{ns_memcached,connect,1},
    {ns_memcached,handle_call,3},
    {gen_server,handle_msg,5},
    {proc_lib,init_p_do_apply,3}

    ]}

=INFO REPORT==== 20-Jun-2010::13:26:59 ===
ns_janitor:current_states: can't reach nodes ['ns_1@127.0.0.1']: trying again

=INFO REPORT==== 20-Jun-2010::13:27:00 ===
ns_memcached connecting to memcached on port 11210

=ERROR REPORT==== 20-Jun-2010::13:27:00 ===
ns_memcached:terminate(badmatch,{error,econnrefused,
[

{ns_memcached,connect,1},
{ns_memcached,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}, {state,undefined})

=ERROR REPORT==== 20-Jun-2010::13:27:02 ===
** Generic server ns_memcached terminating
** Last message in was {stats,"default","vbucket"}
** When Server state == {state,undefined}
** Reason for termination ==
** badmatch,{error,econnrefused,
[{ns_memcached,connect,1}

,

{ns_memcached,handle_call,3}

,

{gen_server,handle_msg,5}

,

{proc_lib,init_p_do_apply,3}

]}