Memcached crashes with Segmentation fault during bucket deletion

Jenkins Ref Link:
http://qa.hq.northscale.net/job/debian_x64--90_04--observe_tests-P0/1/consoleFull

Test Failed:
./testrunner -i yourfile.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,GROUP=ALL -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,sasl_buckets=1,standard_buckets=1,rebalance=out,GROUP=P1

1. Note that failed test will not deterministically reproduce the crash.
2. Same test and some other tests found to be failed on CentOS and Ubuntu but found no core dumps.
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/125/console

[user:info,2014-08-21T5:17:57.340,ns_1@10.3.5.154:<0.750.0>:ns_orchestrator:handle_info:483]Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.32255.122>,
{badmatch,{error,closed,
{gen_server,call,
['ns_memcached-standard_bucket0',

{set_vbucket,610,replica},
180000]}},
{gen_server,call,
[{'janitor_agent-standard_bucket0', 'ns_1@10.3.5.154'},
{if_rebalance,<0.14541.122>,
{dcp_takeover,'ns_1@10.3.5.155',634}},
infinity]}}}}

[error_logger:error,2014-08-21T5:17:57.340,ns_1@10.3.5.154:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:set_initial_vbucket_state/6-fun-0/0
pid: <0.32266.122>
registered_name: []
exception error: {bulk_set_vbucket_state_failed,
[{'ns_1@10.3.5.154',
{'EXIT',
{badmatch,{error,closed,
{gen_server,call,
['ns_memcached-standard_bucket0',
{set_vbucket,610,replica}

,
180000]}},
{gen_server,call,
[

{'janitor_agent-standard_bucket0', 'ns_1@10.3.5.154'}

,
{if_rebalance,<0.14541.122>,
{update_vbucket_state,610,replica,passive,
'ns_1@10.3.5.155'}},
infinity]}}}}]}
in function janitor_agent:bulk_set_vbucket_state/4 (src/janitor_agent.erl, line 411)
ancestors: [<0.32268.122>,<0.14541.122>,<0.14465.122>,<0.750.0>,
mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup,
<0.60.0>]
messages: []
links: [<0.32268.122>]
dictionary: []
trap_exit: false
status: running
heap_size: 2586
stack_size: 27
reductions: 3407
neighbours:
[ns_server:info,2014-08-21T5:17:57.344,ns_1@10.3.5.154:<0.32684.122>:diag_handler:log_all_tap_and_checkpoint_stats:125]logging tap & checkpoint stats
[error_logger:error,2014-08-21T5:17:57.342,ns_1@10.3.5.154:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: ns_memcached:init/1
pid: <0.18706.119>
registered_name: []
exception exit: {badmatch,{error,closed}}
in function gen_server:init_it/6 (gen_server.erl, line 328)
ancestors: ['single_bucket_sup-default',<0.18679.119>]
messages: []
links: [<0.18720.119>,<0.18726.119>,<0.18744.119>,<0.279.0>,
<0.18696.119>]
dictionary: []
trap_exit: true
status: running
heap_size: 75113
stack_size: 27
reductions: 632988
neighbours:
neighbour: [

{pid,<0.18744.119>}

,

{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},
{ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.18679.119>]},
{messages,[]},
{links,[<0.18706.119>,#Port<0.174708>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,6772},
{stack_size,23},
{reductions,525127}]
neighbour: [{pid,<0.18726.119>},
{registered_name,[]}

,
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},

{ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.18679.119>]},
{messages,[]},
{links,[<0.18706.119>,#Port<0.174706>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,28690},
{stack_size,23},
{reductions,3329}]
{reductions,3329}]
neighbour: [{pid,<0.18720.119>},
{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,{gen,do_call,4}},
{ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.18679.119>]}

,

{messages,[]}

,

{links,[<0.18706.119>,#Port<0.174713>]}

,

{dictionary,[]}

,

{trap_exit,false}

,

{status,waiting}

,

{heap_size,46422}

,

{stack_size,23}

,

{reductions,2151481}

]

Please refer attached crash_log.txt.
Core is available at 10.3.5.155:/tmp/backup_crash/21_08_2014_05_45
Uploading Logs