2.0.2 memcached crashes in EventuallyPersistentStore::flushVBucket

http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/196/consoleFull

./testrunner -i /tmp/rebalance_regression.ini wait_timeout=100,get-cbcollect-info=True -t swaprebalance.SwapRebalanceFailedTests.test_failover_swap_rebalance,replica=2,num-buckets=2,num-swap=2,keys-count=1000000,swap-orchestrator=True

test logs:
[2013-03-18 08:35:42,154] - [data_helper:289] INFO - creating direct client 10.3.121.94:11210 bucket-1
[2013-03-18 08:35:42,432] - [rest_client:913] ERROR -

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
[2013-03-18 08:35:42,432] - [rest_client:914] INFO - Latest logs from UI:
[2013-03-18 08:35:42,565] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 1, u'text': u"Node 'ns_1@10.3.121.93' is leaving cluster.", u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1363621308275, u'type': u'info'}

[2013-03-18 08:35:42,567] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 4, u'text': u'Node ns_1@10.3.121.93 left cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1363621307949, u'type': u'info'}

[2013-03-18 08:35:42,567] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.95', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.121.95' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1363621307350, u'type': u'info'}
[2013-03-18 08:35:42,570] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 2, u'text': u"Rebalance exited with reason bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.121.95',\n {'EXIT',\n {{{{unexpected_reason,\n {{badmatch,{error,closed,\n [

{mc_binary,quick_stats_recv,3},\n {mc_binary,\n mass_get_last_closed_checkpoint_loop,\n 5},\n {mc_binary,\n mass_get_last_closed_checkpoint,3},\n {ebucketmigrator_srv,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]}},\n [{misc,executing_on_new_process,1},\n {tap_replication_manager,\n change_vbucket_filter,4},\n {tap_replication_manager,\n '-do_set_incoming_replication_map/3-lc$^5/1-5-',\n 2},\n {tap_replication_manager,\n do_set_incoming_replication_map,3},\n {tap_replication_manager,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n ['tap_replication_manager-bucket-1',\n {change_vbucket_replication,531,\n 'ns_1@10.3.121.96'},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-bucket-1',\n 'ns_1@10.3.121.95'},\n {if_rebalance,<0.13276.97>,\n {update_vbucket_state,531,replica,\n undefined,'ns_1@10.3.121.96'}},\n infinity]}}}},\n {'ns_1@10.3.121.94',\n {'EXIT',\n {{unexpected_reason,\n {{badmatch,{error,closed,\n [{mc_binary,quick_stats_recv,3}

,\n

{mc_binary,quick_stats_loop,5}

,\n

{mc_binary,quick_stats,5}

,\n

{mc_client_binary,\n get_zero_open_checkpoint_vbuckets,3}

,\n

{ebucketmigrator_srv,handle_call,3}

,\n

{gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]}},\n [{misc,executing_on_new_process,1},\n {tap_replication_manager,\n change_vbucket_filter,4},\n {tap_replication_manager,\n '-do_set_incoming_replication_map/3-lc$^5/1-5-',\n 2},\n {tap_replication_manager,\n do_set_incoming_replication_map,3},\n {tap_replication_manager,handle_call,3},\n {gen_server,handle_msg,5}

,\n

{proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n ['tap_replication_manager-bucket-1',\n {change_vbucket_replication,531,\n 'ns_1@10.3.121.95'},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-bucket-1',\n 'ns_1@10.3.121.94'},\n {if_rebalance,<0.13276.97>,\n {update_vbucket_state,531,replica,\n undefined,'ns_1@10.3.121.95'}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4},\n {ns_vbucket_mover,\n update_replication_post_move,3},\n {ns_vbucket_mover,on_move_done,2},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}

]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1363621307292, u'type': u'info'}
[2013-03-18 08:35:42,571] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.95', u'code': 0, u'text': u'Port server memcached on node \'ns_1@10.3.121.95\' exited with status 134. Restarting. Messages: Mon Mar 18 08:41:35.578716 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_451 - Reset vbucket 538 was completed succecssfully.\nMon Mar 18 08:41:35.704327 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_452 - disconnected\nMon Mar 18 08:41:35.805105 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_450"\nMon Mar 18 08:41:35.805157 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_452"\nMon Mar 18 08:41:35.805446 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:35.830322 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 538, cookie 0x6ccf080\nMon Mar 18 08:41:36.347550 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_451 - disconnected\nMon Mar 18 08:41:36.444078 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_453 - Reset vbucket 870 was completed succecssfully.\nMon Mar 18 08:41:36.479457 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_451"\nMon Mar 18 08:41:36.480162 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:36.714685 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 870, cookie 0x6ca6000\nMon Mar 18 08:41:36.976727 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_453 - disconnected\nMon Mar 18 08:41:37.019371 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_454 - Reset vbucket 537 was completed succecssfully.\nMon Mar 18 08:41:37.206467 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_455 - disconnected\nMon Mar 18 08:41:37.271568 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 537, cookie 0x6ca6000\nMon Mar 18 08:41:37.285754 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_453"\nMon Mar 18 08:41:37.285827 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_455"\nMon Mar 18 08:41:37.286268 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:37.643780 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_456 - Reset vbucket 869 was completed succecssfully.\nMon Mar 18 08:41:37.799981 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 869, cookie 0x6cf1b80\nMon Mar 18 08:41:37.837434 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_456 - disconnected\nMon Mar 18 08:41:37.907834 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_454 - disconnected\nMon Mar 18 08:41:37.984234 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_457 - Reset vbucket 868 was completed succecssfully.\nMon Mar 18 08:41:38.035570 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_454"\nMon Mar 18 08:41:38.035669 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_456"\nMon Mar 18 08:41:38.036087 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:38.180437 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_458 - disconnected\nMon Mar 18 08:41:38.250083 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_458"\nMon Mar 18 08:41:38.383886 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:38.544533 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 868, cookie 0x6ca62c0\nMon Mar 18 08:41:38.692906 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_459 - Reset vbucket 536 was completed succecssfully.\nMon Mar 18 08:41:38.933451 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_457 - disconnected\nMon Mar 18 08:41:38.934109 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 536, cookie 0x6ca7080\nMon Mar 18 08:41:39.241149 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_460 - Reset vbucket 867 was completed succecssfully.\nMon Mar 18 08:41:39.396965 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_461 - disconnected\nMon Mar 18 08:41:39.472257 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_457"\nMon Mar 18 08:41:39.472345 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_461"\nMon Mar 18 08:41:39.506632 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:39.553585 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_459 - disconnected\nMon Mar 18 08:41:39.572591 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 867, cookie 0x6ca6000\nMon Mar 18 08:41:39.579114 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_459"\nMon Mar 18 08:41:39.695310 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_460 - disconnected\nMon Mar 18 08:41:39.900652 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_460"\nMon Mar 18 08:41:39.900985 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:40.151512 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_462 - disconnected\nMon Mar 18 08:41:40.266294 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_462"\nMon Mar 18 08:41:40.266461 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:40.856053 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_463 - Reset vbucket 866 was completed succecssfully.\nMon Mar 18 08:41:41.091008 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 866, cookie 0x6ca6000\nMon Mar 18 08:41:41.323419 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_463 - disconnected\nMon Mar 18 08:41:41.342314 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_463"\nMon Mar 18 08:41:41.371404 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_464 - Reset vbucket 535 was completed succecssfully.\nMon Mar 18 08:41:41.661043 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_465 - disconnected\nMon Mar 18 08:41:41.717840 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 535, cookie 0x6ca62c0\nMon Mar 18 08:41:41.742040 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_465"\nMon Mar 18 08:41:41.742448 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:41.954347 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_466 - Reset vbucket 865 was completed succecssfully.\nMon Mar 18 08:41:42.166879 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 865, cookie 0x6cf1600\nMon Mar 18 08:41:42.269048 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_464 - disconnected\nMon Mar 18 08:41:42.340677 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_464"\nMon Mar 18 08:41:42.340711 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:42.384967 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_467 - Reset vbucket 864 was completed succecssfully.\nMon Mar 18 08:41:42.413198 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_466 - disconnected\nMon Mar 18 08:41:42.441804 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_466"\nMon Mar 18 08:41:42.684453 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 864, cookie 0x6ccf080\nMon Mar 18 08:41:42.818667 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_468 - disconnected\nMon Mar 18 08:41:42.875395 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_467 - disconnected\nMon Mar 18 08:41:42.904356 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_469 - Reset vbucket 534 was completed succecssfully.\nMon Mar 18 08:41:42.978903 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_467"\nMon Mar 18 08:41:42.978974 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_468"\nMon Mar 18 08:41:42.979235 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:43.243646 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_470 - disconnected\nMon Mar 18 08:41:43.403811 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:43.403750 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_470"\nMon Mar 18 08:41:43.494580 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 534, cookie 0x6ccf600\nMon Mar 18 08:41:43.580734 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_471 - Reset vbucket 533 was completed succecssfully.\nMon Mar 18 08:41:43.893154 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 533, cookie 0x6ca6000\nMon Mar 18 08:41:44.327385 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_472 - Reset vbucket 532 was completed succecssfully.\nMon Mar 18 08:41:44.412144 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_469 - disconnected\nMon Mar 18 08:41:44.495170 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_469"\nMon Mar 18 08:41:44.502353 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_471 - disconnected\nMon Mar 18 08:41:44.516347 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_471"\nMon Mar 18 08:41:44.516369 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:44.519367 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 532, cookie 0x6ca62c0\nMon Mar 18 08:41:44.855524 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:45.087666 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_473 - Reset vbucket 863 was completed succecssfully.\nMon Mar 18 08:41:45.320777 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_472 - disconnected\nMon Mar 18 08:41:45.440897 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 863, cookie 0x6cceb00\nMon Mar 18 08:41:45.445425 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_472"\nMon Mar 18 08:41:45.446922 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:45.531819 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_474 - Reset vbucket 531 was completed succecssfully.\nMon Mar 18 08:41:45.566986 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_473 - disconnected\nMon Mar 18 08:41:45.641574 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_473"\nMon Mar 18 08:41:45.822884 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 531, cookie 0x6ccf600\nMon Mar 18 08:41:45.850503 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_475 - disconnected\nMon Mar 18 08:41:45.932729 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_475"\nMon Mar 18 08:41:45.932798 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:46.151290 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_476 - Reset vbucket 530 was completed succecssfully.\nMon Mar 18 08:41:46.393310 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_474 - disconnected\nmemcached: src/ep.cc:1790: virtual void PersistenceCallback::callback(mutation_result&): Assertion `stats->diskQueueSize < ((size_t)1<<(sizeof(size_t)*8-1))\' failed.', u'shortText': u'message', u'module': u'ns_port_server', u'tstamp': 1363621307102, u'type': u'info'}

[2013-03-18 08:35:42,577] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'Bucket "bucket-1" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1363620900761, u'type': u'info'}

[2013-03-18 08:35:42,578] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.95', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.121.95\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1363620899668, u'type': u'info'}

[2013-03-18 08:35:42,579] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.96', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.121.96\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1363620899353, u'type': u'info'}

[2013-03-18 08:35:42,580] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.94', u'code': 5, u'text': u"Node 'ns_1@10.3.121.94' saw that node 'ns_1@10.3.121.98' went down.", u'shortText': u'node down', u'module': u'ns_node_disco', u'tstamp': 1363620899198, u'type': u'warning'}

[2013-03-18 08:35:42,585] - [rest_client:915] ERROR -

{u'node': u'ns_1@10.3.121.95', u'code': 5, u'text': u"Node 'ns_1@10.3.121.95' saw that node 'ns_1@10.3.121.98' went down.", u'shortText': u'node down', u'module': u'ns_node_disco', u'tstamp': 1363620899195, u'type': u'warning'}

ERROR

root@ubuntu1104-64:/tmp# sudo gdb /opt/couchbase/bin/memcached core.memcached.30589
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 30607]
[New Thread 30597]
[New Thread 30611]
[New Thread 30589]
[New Thread 30604]
[New Thread 30598]
[New Thread 30602]
[New Thread 30600]
[New Thread 30603]
[New Thread 30601]
[New Thread 30606]
[New Thread 30610]
[New Thread 30609]
[New Thread 30608]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007f9d01257d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) t a a bt

Thread 14 (Thread 30608):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca013d8 in wait (this=0x24ee2d0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x24ee2d0, d=...) at src/dispatcher.cc:328
#3 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9b880) at src/dispatcher.cc:171
#4 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x6d9b8d4) at src/dispatcher.cc:28
#5 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 13 (Thread 30609):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca013d8 in wait (this=0x24ee240, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x24ee240, d=...) at src/dispatcher.cc:328
#3 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9b6c0) at src/dispatcher.cc:171
#4 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x6d9b714) at src/dispatcher.cc:28
#5 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 12 (Thread 30610):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca013d8 in wait (this=0x24ee5a0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x24ee5a0, d=...) at src/dispatcher.cc:328
#3 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9b500) at src/dispatcher.cc:171
#4 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x6d9b554) at src/dispatcher.cc:28
#5 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 11 (Thread 30606):
#0 0x00007f9d012d44ed in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d01305914 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9cfca416a5 in updateStatsThread (arg=<value optimized out>) at src/memory_tracker.cc:31
#3 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 10 (Thread 30601):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4e280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4e280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x24e94f8) at daemon/thread.c:301
#4 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 9 (Thread 30603):
#0 mc_get_allocation_size (ptr=0x7977200) at daemon/alloc_hooks.c:115
#1 0x00007f9cfca41264 in DeleteHook (ptr=0x7977200) at src/memory_tracker.cc:56
#2 0x00007f9d01801402 in MallocHook::InvokeDeleteHookSlow(void const*) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#3 0x00007f9d017f455a in MallocHook::InvokeDeleteHook(void const*) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#4 0x00007f9d018077f6 in tc_delete () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#5 0x00007f9d00fb3e19 in std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_stringstream() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
--Type <return> to continue, or q <return> to quit--
#6 0x00007f9cfca3809a in void core_engine::add_casted_stat<unsigned long>(char const*, unsigned long, void (char const*, unsigned short, char const*, unsigned int, void const*), void const*) ()
from /opt/couchbase/lib/memcached/ep.so

#7 0x00007f9cfca28af0 in addCheckpointStat (this=<value optimized out>, vb=...) at src/ep_engine.cc:2858
#8 EventuallyPersistentEngine::StatCheckpointVisitor::visitBucket (this=<value optimized out>, vb=...) at src/ep_engine.cc:2842

#9 0x00007f9cfca09adc in EventuallyPersistentStore::visit (this=<value optimized out>, visitor=...) at src/ep.cc:2417
#10 0x00007f9cfca2d315 in EventuallyPersistentEngine::doCheckpointStats (this=0x6da2000, cookie=0x6cf18c0, add_stat=0x40d1d0 <append_stats>, stat_key=<value optimized out>, nkey=<value optimized out>)
at src/ep_engine.cc:2868
#11 0x00007f9cfca2fdcc in EventuallyPersistentEngine::getStats (this=0x6da2000, cookie=0x6cf18c0, stat_key=0x6cf5018 "checkpointn_ns_1@10.3.121.94", nkey=10, add_stat=0x40d1d0 <append_stats>)
at src/ep_engine.cc:3328
#12 0x00007f9cfca30416 in EvpGetStats (handle=0x6da2000, cookie=0x6cf18c0, stat_key=0x6cf5018 "checkpointn_ns_1@10.3.121.94", nkey=10, add_stat=<value optimized out>) at src/ep_engine.cc:193
#13 0x00007f9cff4e1c20 in bucket_get_stats (handle=<value optimized out>, cookie=0x6cf18c0, stat_key=0x6cf5018 "checkpointn_ns_1@10.3.121.94", nkey=10, add_stat=0x40d1d0 <append_stats>)
at bucket_engine.c:1720
#14 0x00000000004113f6 in process_bin_stat (c=0x6cf18c0) at daemon/memcached.c:2199
#15 0x0000000000411d65 in complete_nread_binary (c=0x7977200) at daemon/memcached.c:3708
#16 complete_nread (c=0x7977200) at daemon/memcached.c:3820
#17 conn_nread (c=0x7977200) at daemon/memcached.c:5673
#18 0x00000000004068a5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x6cf18c0) at daemon/memcached.c:5936
#19 0x00007f9d020c748c in event_process_active_single_queue (base=0x6d4ea00, flags=<value optimized out>) at event.c:1308
#20 event_process_active (base=0x6d4ea00, flags=<value optimized out>) at event.c:1375
#21 event_base_loop (base=0x6d4ea00, flags=<value optimized out>) at event.c:1572
#22 0x0000000000414b94 in worker_libevent (arg=0x24e96e8) at daemon/thread.c:301
#23 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#24 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#25 0x0000000000000000 in ?? ()

Thread 8 (Thread 30600):
#0 0x00007f9d015ca955 in __lll_unlock_wake () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9d015c6e7a in _L_unlock_1177 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f9d015c6da3 in pthread_mutex_unlock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f9cfca42636 in Mutex::release (this=0xed948a8) at src/mutex.cc:94
#4 0x00007f9cfc9fab10 in unlock (this=<value optimized out>, qi=..., vbucket=...) at src/locks.hh:58
#5 ~LockHolder (this=<value optimized out>, qi=..., vbucket=...) at src/locks.hh:41
#6 CheckpointManager::queueDirty (this=<value optimized out>, qi=..., vbucket=...) at src/checkpoint.cc:717
#7 0x00007f9cfca07423 in EventuallyPersistentStore::queueDirty (this=0x6d9e480, vb=..., key=..., vbid=<value optimized out>, op=queue_op_set, seqno=<value optimized out>, tapBackfill=false)
at src/ep.cc:2168
#8 0x00007f9cfca0bc2d in EventuallyPersistentStore::setWithMeta (this=0x6d9e480, itm=..., cas=<value optimized out>, cookie=<value optimized out>, force=<value optimized out>,
allowExisting=<value optimized out>, nru=3 '\003') at src/ep.cc:1399
#9 0x00007f9cfca2ac53 in EventuallyPersistentEngine::tapNotify(const void *, void *, uint16_t, uint8_t, uint16_t, <anonymous enum>, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, const void *, size_t, uint16_t) (this=0x6da2000, cookie=0x6ccf080, engine_specific=<value optimized out>, nengine=<value optimized out>, tap_flags=<value optimized out>, tap_event=TAP_MUTATION, tap_seqno=974,
key=0x1102c031, nkey=25, flags=0, exptime=0, cas=15671462831700303, data=0x1102c04a, ndata=369, vbucket=530) at src/ep_engine.cc:2067
#10 0x00007f9cfca2b368 in EvpTapNotify(ENGINE_HANDLE *, const void *, void *, uint16_t, uint8_t, uint16_t, <anonymous enum>, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, const void *, size_t, uint16_t) (handle=0x6da2000, cookie=0x6ccf080, engine_specific=0x1102c028, nengine=65535, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=974, key=0x1102c031, nkey=25, flags=0,
exptime=0, cas=15671462831700303, data=0x1102c04a, ndata=369, vbucket=<value optimized out>) at src/ep_engine.cc:1037
#11 0x00007f9cff4e0a04 in bucket_tap_notify (handle=<value optimized out>, cookie=0x6ccf080, engine_specific=0x1102c028, nengine=65535, ttl=254 '\376', tap_flags=23, tap_event=TAP_MUTATION,
tap_seqno=974, key=0x1102c031, nkey=25, flags=0, exptime=0, cas=15671462831700303, data=0x1102c04a, ndata=369, vbucket=<value optimized out>) at bucket_engine.c:1942
#12 0x0000000000409eb2 in process_bin_tap_packet (event=TAP_MUTATION, c=0x6ccf080) at daemon/memcached.c:3031
#13 0x00000000004120c3 in process_bin_packet (c=0x6ccf080) at daemon/memcached.c:3117
#14 complete_nread_binary (c=0x6ccf080) at daemon/memcached.c:3738
#15 complete_nread (c=0x6ccf080) at daemon/memcached.c:3820
#16 conn_nread (c=0x6ccf080) at daemon/memcached.c:5673
#17 0x00000000004068a5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x6ccf080) at daemon/memcached.c:5936
#18 0x00007f9d020c748c in event_process_active_single_queue (base=0x6d4e500, flags=<value optimized out>) at event.c:1308
#19 event_process_active (base=0x6d4e500, flags=<value optimized out>) at event.c:1375
#20 event_base_loop (base=0x6d4e500, flags=<value optimized out>) at event.c:1572
#21 0x0000000000414b94 in worker_libevent (arg=0x24e9400) at daemon/thread.c:301
--Type <return> to continue, or q <return> to quit--
#22 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#23 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#24 0x0000000000000000 in ?? ()

Thread 7 (Thread 30602):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4ec80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4ec80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x24e95f0) at daemon/thread.c:301
#4 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 6 (Thread 30598):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9d00102176 in logger_thead_main (arg=<value optimized out>) at extensions/loggers/file_logger.c:368
#2 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

Thread 5 (Thread 30604):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4e780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4e780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x24e97e0) at daemon/thread.c:301
#4 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 4 (Thread 30589):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4e000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4e000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040c2e1 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918

Thread 3 (Thread 30611):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca2090f in wait (this=0x6da2000) at src/syncobject.hh:57
#2 wait (this=0x6da2000) at src/syncobject.hh:73
#3 wait (this=0x6da2000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x6da2000) at src/ep_engine.cc:3406
#5 0x00007f9cfca209f3 in EvpNotifyPendingConns (arg=0x6da2000) at src/ep_engine.cc:1139
#6 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 2 (Thread 30597):
#0 0x00007f9d012fe2ed in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d01299798 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9d0129a7be in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f9d0128e8fa in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f9d0128d7ca in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f9d00b05b19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#7 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#8 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 1 (Thread 30607):
#0 0x00007f9d01257d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d0125bab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9d012507c5 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f9cfca1d8db in PersistenceCallback::callback(std::pair<int, long>&) () from /opt/couchbase/lib/memcached/ep.so
#4 0x00007f9cfca78724 in CouchKVStore::commitCallback(CouchRequest **, int, <anonymous enum>) (this=0x6dbc000, committedReqs=<value optimized out>, numReqs=10, errCode=COUCHSTORE_SUCCESS)
at src/couch-kvstore/couch-kvstore.cc:1655
#5 0x00007f9cfca7c07c in CouchKVStore::commit2couchstore (this=0x6dbc000) at src/couch-kvstore/couch-kvstore.cc:1488
#6 0x00007f9cfca7c25a in CouchKVStore::commit (this=0x777d) at src/couch-kvstore/couch-kvstore.cc:871
#7 0x00007f9cfca0ee06 in EventuallyPersistentStore::flushVBucket (this=0x6d9e480, vbid=<value optimized out>) at src/ep.cc:1977
#8 0x00007f9cfca3ca1a in doFlush (this=0x6dba5a0, d=..., tid=...) at src/flusher.cc:215
#9 Flusher::step (this=0x6dba5a0, d=..., tid=...) at src/flusher.cc:153
#10 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9aa80) at src/dispatcher.cc:171
#11 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x777d) at src/dispatcher.cc:28
#12 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x0000000000000000 in ?? ()