memcached segfaults during rebalance

I've seen this a couple times now. I was running membase commit 31699e15592fb6923977729f33f8f9b8b9350ef1, memcached 48576af406ee43ae7c3b3f37395cac205ad375bb.

I had no data stored. I suspect a race between deleting and recreating vbuckets.

Here's the backtrace:

Process: memcached [1941]
Path: /Users/sean/northscale/memcached/memcached
Identifier: memcached
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: beam.smp [1886]

Date/Time: 2010-06-20 15:03:04.733 -0700
OS Version: Mac OS X 10.6.3 (10D573)
Report Version: 6

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000008
Crashed Thread: 9

Thread 0: Dispatch queue: com.apple.main-thread
0 libSystem.B.dylib 0x00007fff80d824ea kevent + 10
1 libevent-1.4.2.dylib 0x00000001000368a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x0000000100027d08 event_base_loop + 664
3 memcached 0x0000000100004cf4 main + 4212 (memcached.c:6492)
4 memcached 0x0000000100000cd4 start + 52

Thread 1:
0 libSystem.B.dylib 0x00007fff80d6b362 read$NOCANCEL + 10
1 libSystem.B.dylib 0x00007fff80dba8be __sread + 16
2 libSystem.B.dylib 0x00007fff80db2604 _sread + 19
3 libSystem.B.dylib 0x00007fff80db249a __srefill + 41
4 libSystem.B.dylib 0x00007fff80dbd65a __srget + 17
5 libSystem.B.dylib 0x00007fff80dbd616 getc + 63
6 stdin_term_handler.0.so 0x0000000100061dd8 check_stdin_thread + 40 (stdin_check.c:17)
7 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
8 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 2:
0 libSystem.B.dylib 0x00007fff80da4316 __semwait_signal + 10
1 libSystem.B.dylib 0x00007fff80da8131 _pthread_cond_wait + 1286
2 ep.0.so 0x000000010021f3bc SyncObject::wait(timeval const&) + 72 (syncobject.hh:42)
3 ep.0.so 0x0000000100202082 Dispatcher::run() + 388 (dispatcher.cc:49)
4 ep.0.so 0x0000000100202537 launch_dispatcher_thread + 30 (dispatcher.cc:10)
5 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
6 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 3:
0 libSystem.B.dylib 0x00007fff80da4316 __semwait_signal + 10
1 libSystem.B.dylib 0x00007fff80da8131 _pthread_cond_wait + 1286
2 ep.0.so 0x000000010021f3bc SyncObject::wait(timeval const&) + 72 (syncobject.hh:42)
3 ep.0.so 0x000000010021f53c SyncObject::wait(double) + 58 (syncobject.hh:58)
4 ep.0.so 0x000000010021f752 EventuallyPersistentEngine::notifyTapIoThreadMain() + 528 (ep_engine.h:1322)
5 ep.0.so 0x000000010021f8bd EventuallyPersistentEngine::notifyTapIoThread() + 33 (ep_engine.h:1353)
6 ep.0.so 0x000000010021332d EvpNotifyTapIo + 21 (ep_engine.cc:561)
7 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
8 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 4:
0 libSystem.B.dylib 0x00007fff80d824ea kevent + 10
1 libevent-1.4.2.dylib 0x00000001000368a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x0000000100027d08 event_base_loop + 664
3 memcached 0x0000000100010471 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 5:
0 libSystem.B.dylib 0x00007fff80d824ea kevent + 10
1 libevent-1.4.2.dylib 0x00000001000368a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x0000000100027d08 event_base_loop + 664
3 memcached 0x0000000100010471 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 6:
0 libSystem.B.dylib 0x00007fff80d824ea kevent + 10
1 libevent-1.4.2.dylib 0x00000001000368a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x0000000100027d08 event_base_loop + 664
3 memcached 0x0000000100010471 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 7:
0 libSystem.B.dylib 0x00007fff80d824ea kevent + 10
1 libevent-1.4.2.dylib 0x00000001000368a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x0000000100027d08 event_base_loop + 664
3 memcached 0x0000000100010471 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 8:
0 libSystem.B.dylib 0x00007fff80d824ea kevent + 10
1 libevent-1.4.2.dylib 0x00000001000368a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x0000000100027d08 event_base_loop + 664
3 memcached 0x0000000100010471 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 9 Crashed:
0 ep.0.so 0x000000010022bea8 HashTable::visit(HashTableVisitor&) + 174 (stored-value.cc:79)
1 ep.0.so 0x000000010021d478 EventuallyPersistentStore::visit(VBucketVisitor&) + 208 (ep.hh:161)
2 ep.0.so 0x000000010021294d launch_backfill_thread + 64 (ep_engine.cc:661)
3 libSystem.B.dylib 0x00007fff80da28b6 _pthread_start + 331
4 libSystem.B.dylib 0x00007fff80da2769 thread_start + 13

Thread 9 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000008 rbx: 0x000000010010e480 rcx: 0x0000000000000002 rdx: 0x000000010010e480
rdi: 0x0000000000000008 rsi: 0x000000010010e480 rbp: 0x0000000101986e40 rsp: 0x0000000101986df0
r8: 0x0000000102c5ebf8 r9: 0x0000000100112950 r10: 0x0000000000000000 r11: 0x000000000000001f
r12: 0x0000000000000000 r13: 0x0000000000002913 r14: 0x000000010021290d r15: 0x000000010010e480
rip: 0x000000010022bea8 rfl: 0x0000000000010206 cr2: 0x0000000000000008

Binary Images:
0x100000000 - 0x10001afef +memcached ??? (???) <9775B2DA-4E38-C960-C783-2AAAD2DFE618> /Users/sean/northscale/ns_server/bin/memcached/memcached
0x100026000 - 0x10003cfef +libevent-1.4.2.dylib 4.3.0 (compatibility 4.0.0) <F95EF8E0-1F7F-4754-4837-3621784ECE06> /usr/local/homebrew/Cellar/libevent/1.4.13/lib/libevent-1.4.2.dylib
0x100061000 - 0x100061ff7 +stdin_term_handler.0.so ??? (???) <4DAE0248-460C-95A6-797B-2FDD05D4C101> /Users/sean/northscale/memcached/.libs/stdin_term_handler.0.so
0x100200000 - 0x1002e5fef +ep.0.so ??? (???) <DB7002E8-0088-E352-A8DF-39200D17BB86> /Users/sean/northscale/membase/.libs/ep.0.so
0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???) <B633F790-4DDB-53CD-7ACF-2A3682BCEA9F> /usr/lib/dyld
0x7fff80d68000 - 0x7fff80f27ff7 libSystem.B.dylib 125.0.1 (compatibility 1.0.0) <CB9A4929-61AF-DE71-5635-133E9EC95783> /usr/lib/libSystem.B.dylib
0x7fff8392e000 - 0x7fff8394ffff libresolv.9.dylib 40.0.0 (compatibility 1.0.0) <1AE68BBB-6536-125C-DE2A-13CA916D0EC4> /usr/lib/libresolv.9.dylib
0x7fff84c4b000 - 0x7fff84c4fff7 libmathCommon.A.dylib 315.0.0 (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> /usr/lib/system/libmathCommon.A.dylib
0x7fff88bd9000 - 0x7fff88c56fef libstdc+.6.dylib 7.9.0 (compatibility 7.0.0) <35ECA411-2C08-FD7D-11B1-1B7A04921A5C> /usr/lib/libstdc+.6.dylib
0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???) <CB9A4929-61AF-DE71-5635-133E9EC95783> /usr/lib/libSystem.B.dylib