[system test] memcached crashed during reblance

4 node ubuntu 11.04 64bit install couchbase server 1.8.1-937
1 node ubuntu 11.04 64bit install couchbase server 2.0.1-168
Do online upgrade by swap rebalance 2.0.1 node with 1.8.1 node.
Rebalance failed due to memcached of one 1.8.1 node crashed as stack trace below.

I will check if this is the same crash we saw in 1.8.1 tests.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 couchbase couchbase 1670307 2012-06-27 00:36 /opt/couchbase/bin/memcached
-rwxrwxrwx 1 ubuntu ubuntu 9384378368 2013-02-27 03:24 /tmp/core.memcached.2341
21225adeb5ee5826ffec6b2b6cc68357 /opt/couchbase/bin/memcached
eaaa93ffd8ca1e528028184af6fefe6e /tmp/core.memcached.2341
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 2364]
[New Thread 2366]
[New Thread 2341]
[New Thread 2353]
[New Thread 2351]
[New Thread 2359]
[New Thread 2354]
[New Thread 2358]
[New Thread 2360]
[New Thread 2361]
[New Thread 2362]
[New Thread 2355]
[New Thread 2350]
[New Thread 2352]
[New Thread 2367]
[New Thread 2365]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007f5255c55d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6

Thread 16 (Thread 2365):
#0 0x00007f5255d0343d in fdatasync () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f525181d259 in full_fsync (id=0x195c4f8, flags=2) at embedded/sqlite3.c:25510
#2 unixSync (id=0x195c4f8, flags=2) at embedded/sqlite3.c:25558
#3 0x00007f5251816547 in vfsepstatSync (pFile=0x195c4d0, flags=2) at sqlite-vfs.c:160
#4 0x00007f5251838783 in sqlite3OsSync (pWal=0x18f1508, sync_flags=2, nBuf=-1, zBuf=<value optimized out>) at embedded/sqlit
e3.c:13413
#5 walCheckpoint (pWal=0x18f1508, sync_flags=2, nBuf=-1, zBuf=<value optimized out>) at embedded/sqlite3.c:42447
#6 sqlite3WalCheckpoint (pWal=0x18f1508, sync_flags=2, nBuf=-1, zBuf=<value optimized out>) at embedded/sqlite3.c:43431
#7 0x00007f52518548a2 in sqlite3PagerCheckpoint (db=0x195ce08, zDb=<value optimized out>) at embedded/sqlite3.c:40675
#8 sqlite3BtreeCheckpoint (db=0x195ce08, zDb=<value optimized out>) at embedded/sqlite3.c:52433
#9 sqlite3Checkpoint (db=0x195ce08, zDb=<value optimized out>) at embedded/sqlite3.c:39117
#10 sqlite3_wal_checkpoint (db=0x195ce08, zDb=<value optimized out>) at embedded/sqlite3.c:39082
#11 0x00007f5251854adb in sqlite3WalDefaultHook (pClientData=<value optimized out>, db=0x195ce08, zDb=0x1 <Address 0x1 out of
bounds>, nFrame=-1) at embedded/sqlite3.c:104546
#12 0x00007f525189ae0c in doWalCallbacks (pStmt=0x212314908) at embedded/sqlite3.c:57877
#13 sqlite3Step (pStmt=0x212314908) at embedded/sqlite3.c:57962
#14 sqlite3_step (pStmt=0x212314908) at embedded/sqlite3.c:58011
#15 0x00007f525180b7ca in PreparedStatement::execute (this=0x7f524ef6f8c0) at sqlite-pst.cc:73
#16 0x00007f5251809e33 in execute (this=0x4b801b0) at sqlite-kvstore.hh:168
#17 StrategicSqlite3::commit (this=0x4b801b0) at sqlite-kvstore.hh:74
#18 0x00007f52517a35ed in TransactionContext::commit (this=0x628ab98) at ep.cc:2205
#19 0x00007f52517ad35f in EventuallyPersistentStore::flushSome (this=0x628aa00, q=0x628ab30, rejectQueue=0x7dfb360) at ep.cc:
1528
#20 0x00007f52517da4de in Flusher::doFlush (this=0x6392360) at flusher.cc:242
#21 0x00007f52517daf1d in Flusher::step (this=0x57, d=..., tid=DWARF-2 expression error: DW_OP_reg operations must be used ei
ther alone or in conjuction with DW_OP_piece or DW_OP_bit_piece.
) at flusher.cc:156
#22 0x00007f52517db5d0 in FlusherStepper::callback (this=<value optimized out>, d=..., t=<value optimized out>) at flusher.cc
:23
#23 0x00007f52517a19ef in Task::run(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#24 0x00007f52517a0981 in Dispatcher::run (this=0x1908c40) at dispatcher.cc:123
#25 0x00007f52517a10db in launch_dispatcher_thread (arg=0x57) at dispatcher.cc:28
#26 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#27 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#28 0x0000000000000000 in ?? ()

Thread 15 (Thread 2367):
#0 0x00007f5255fc87c4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f5255fc35b4 in _L_lock_944 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f5255fc33ca in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x0000000000415a26 in notify_io_complete (cookie=0x7482dc8, status=ENGINE_SUCCESS) at daemon/thread.c:485
#4 0x00007f52518031f9 in operator() (this=<value optimized out>) at /usr/include/c++/4.4/bits/stl_function.h:457
#5 operator() (this=<value optimized out>) at /usr/include/c++/4.4/backward/binders.h:153
#6 for_each<std::_List_iterator<void const*>, std::binder2nd<std::pointer_to_binary_function<void const*, ENGINE_ERROR_CODE,
void> > > (this=<value optimized out>) at /usr/include/c++/4.4/bits/stl_algo.h:4200
#7 notifyIOComplete<std::list<void const*, std::allocator<void const*> > > (this=<value optimized out>) at ep_engine.h:434
#8 TapConnMap::notifyIOThreadMain (this=<value optimized out>) at tapconnmap.cc:467
#9 0x00007f52517bc85f in EventuallyPersistentEngine::notifyPendingConnections (this=0x4cf6000) at ep_engine.cc:3911
#10 0x00007f52517bc9e3 in EvpNotifyPendingConns (arg=0x4cf6000) at ep_engine.cc:1088
#11 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()

Thread 14 (Thread 2352):
#0 0x00007f5255d0b633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5256ad9f36 in epoll_dispatch (base=0x18ec280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f5256ac5394 in event_base_loop (base=0x18ec280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414d14 in worker_libevent (arg=0x18c45f8) at daemon/thread.c:301
#4 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 13 (Thread 2350):
#0 0x00007f5255cfc2ed in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5255c97798 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f5255c987be in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f5255c8c8fa in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f5255c8b7ca in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f5255503b19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#7 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 12 (Thread 2355):
#0 0x00007f5255d0b633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5256ad9f36 in epoll_dispatch (base=0x18ec780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f5256ac5394 in event_base_loop (base=0x18ec780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414d14 in worker_libevent (arg=0x18c48e0) at daemon/thread.c:301
#4 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 11 (Thread 2362):
#0 0x00007f5255fc5f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f52517bc8f1 in wait (this=0x18e8000) at syncobject.hh:42
#2 wait (this=0x18e8000) at syncobject.hh:58
#3 wait (this=0x18e8000) at tapconnmap.hh:171
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x18e8000) at ep_engine.cc:3918
#5 0x00007f52517bc9e3 in EvpNotifyPendingConns (arg=0x18e8000) at ep_engine.cc:1088
#6 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 10 (Thread 2361):
#0 0x00007f5255fc5f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f525179d64a in wait (this=0x197a980, d=...) at syncobject.hh:42
#2 IdleTask::run (this=0x197a980, d=...) at dispatcher.cc:286
#3 0x00007f52517a0981 in Dispatcher::run (this=0x1908a80) at dispatcher.cc:123
#4 0x00007f52517a10db in launch_dispatcher_thread (arg=0x1908ad4) at dispatcher.cc:28
#5 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 9 (Thread 2360):
#0 0x00007f5255fc8cbd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f525181d376 in read (id=0x1991578, pBuf=0x1427ac050, amt=1024, offset=277950464) at /usr/include/bits/unistd.h:45
#2 seekAndRead (id=0x1991578, pBuf=0x1427ac050, amt=1024, offset=277950464) at embedded/sqlite3.c:25247
#3 unixRead (id=0x1991578, pBuf=0x1427ac050, amt=1024, offset=277950464) at embedded/sqlite3.c:25281
#4 0x00007f525181666e in vfsepstatRead (pFile=0x1991550, zBuf=0x1427ac050, iAmt=1024, iOfst=277950464) at sqlite-vfs.c:112
#5 0x00007f52518379fa in sqlite3OsRead (pPg=0x1427ac008) at embedded/sqlite3.c:13402
#6 readDbPage (pPg=0x1427ac008) at embedded/sqlite3.c:36959
#7 0x00007f525183cc54 in sqlite3PagerAcquire (pPager=0x1991408, pgno=271437, ppPage=0x7f5250f73128, noContent=0) at embedded
/sqlite3.c:39100
#8 0x00007f52518469bc in btreeGetPage (pCur=0x1af169e90, newPgno=1115340880) at embedded/sqlite3.c:46084
#9 getAndInitPage (pCur=0x1af169e90, newPgno=1115340880) at embedded/sqlite3.c:46137
#10 moveToChild (pCur=0x1af169e90, newPgno=1115340880) at embedded/sqlite3.c:48599
#11 0x00007f5251858f96 in sqlite3BtreeMovetoUnpacked (pCur=0x1af169e90, pIdxKey=<value optimized out>, intKey=<value optimize
d out>, biasRight=<value optimized out>, pRes=<value optimized out>) at embedded/sqlite3.c:49059
#12 0x00007f525188fb49 in sqlite3VdbeExec (p=<value optimized out>) at embedded/sqlite3.c:63099
#13 0x00007f525189a8e0 in sqlite3Step (pStmt=0x1e50308) at embedded/sqlite3.c:57947
#14 sqlite3_step (pStmt=0x1e50308) at embedded/sqlite3.c:58011
#15 0x00007f525180b7ca in PreparedStatement::execute (this=0x1e44250) at sqlite-pst.cc:73
#16 0x00007f5251807cc9 in StrategicSqlite3::del (this=0x18ce4b0, key=..., rowid=2831, vb=<value optimized out>, vbver=<value
optimized out>, cb=...) at sqlite-kvstore.cc:211
#17 0x00007f52517acbbd in EventuallyPersistentStore::flushOneDelOrSet (this=0x2c46280, qi=<value optimized out>, rejectQueue=
<value optimized out>) at ep.cc:1938
#18 0x00007f52517ad25e in EventuallyPersistentStore::flushOne (this=0x2c46280, q=0x2c463b0, rejectQueue=0x7f80bf90) at ep.cc:
1974
#19 0x00007f52517ad2fe in EventuallyPersistentStore::flushSome (this=0x2c46280, q=0x2c463b0, rejectQueue=0x7f80bf90) at ep.cc
:1520
#20 0x00007f52517da4de in Flusher::doFlush (this=0x2d63560) at flusher.cc:242
#21 0x00007f52517daf1d in Flusher::step (this=0x2f, d=..., tid=DWARF-2 expression error: DW_OP_reg operations must be used ei
ther alone or in conjuction with DW_OP_piece or DW_OP_bit_piece.
) at flusher.cc:156
#22 0x00007f52517db5d0 in FlusherStepper::callback (this=<value optimized out>, d=..., t=<value optimized out>) at flusher.cc
:23
#23 0x00007f52517a19ef in Task::run(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#24 0x00007f52517a0981 in Dispatcher::run (this=0x1908540) at dispatcher.cc:123
#25 0x00007f52517a10db in launch_dispatcher_thread (arg=0x2f) at dispatcher.cc:28
#26 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#27 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#28 0x0000000000000000 in ?? ()

Thread 8 (Thread 2358):
#0 0x00007f5255cd24ed in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5255d03914 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f52517e1325 in updateStatsThread (arg=<value optimized out>) at memory_tracker.cc:31
#3 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 7 (Thread 2354):
#0 0x00007f5255fc87c4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f5255fc35b4 in _L_lock_944 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f5255fc33ca in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f52517915ca in acquire (this=0x7f52532f2a40, m=...) at mutex.hh:70
#4 lock (this=0x7f52532f2a40, m=...) at locks.hh:48
#5 LockHolder::LockHolder (this=0x7f52532f2a40, m=...) at locks.hh:26
#6 0x00007f52517d63ff in EventuallyPersistentEngine::doWalkTapQueue(void const*, void*, void, unsigned short, unsigned c
har*, unsigned short*, unsigned int*, unsigned short*, TapProducer*, bool&) () from /opt/couchbase/lib/memcached/ep.so
#7 0x00007f52517c769f in EventuallyPersistentEngine::walkTapQueue (this=0x4cf6000, cookie=0x1f8b09b88, itm=<value optimized
out>, es=<value optimized out>, nes=<value optimized out>, ttl=<value optimized out>, flags=0x7f52532f2d0a, seqno=0x7f52532f2
d04, vbucket=0x7f52532f2d08) at ep_engine.cc:2242
#8 0x00007f52517c77ed in EvpTapIterator (handle=0x4cf6000, cookie=0x1f8b09b88, itm=0x7f52532f2cf8, es=0x7f52532f2cf0, nes=<v
alue optimized out>, ttl=<value optimized out>, flags=0x7f52532f2d0a, seqno=0x7f52532f2d04, vbucket=0x7f52532f2d08) at ep_eng
ine.cc:997
#9 0x00007f5254afcfb9 in bucket_tap_iterator_shim (handle=0x7f5254d01180, cookie=0x1f8b09b88, itm=0x7f52532f2cf8, engine_spe
cific=0x7f52532f2cf0, nengine_specific=<value optimized out>, ttl=<value optimized out>, flags=0x7f52532f2d0a, seqno=0x7f5253
2f2d04, vbucket=0x7f52532f2d08) at bucket_engine.c:1876
#10 0x00000000004136ce in ship_tap_log (c=0x1f8b09b88) at daemon/memcached.c:2381
#11 conn_ship_log (c=0x1f8b09b88) at daemon/memcached.c:5197
#12 0x0000000000408875 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x1f8b09b88) at daemon/me
mcached.c:5610
#13 0x00007f5256ac548c in event_process_active_single_queue (base=0x18eca00, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x18eca00, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x18eca00, flags=<value optimized out>) at event.c:1572
#16 0x0000000000414d14 in worker_libevent (arg=0x18c47e8) at daemon/thread.c:301
#17 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#19 0x0000000000000000 in ?? ()

Thread 6 (Thread 2359):
#0 0x00007f5255fc5bac in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f52517a0bb9 in wait (this=0x1908380) at syncobject.hh:31
#2 Dispatcher::run (this=0x1908380) at dispatcher.cc:89
#3 0x00007f52517a10db in launch_dispatcher_thread (arg=0x19083d4) at dispatcher.cc:28
#4 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 2351):
#0 0x00007f5255d0b633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5256ad9f36 in epoll_dispatch (base=0x18ec500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f5256ac5394 in event_base_loop (base=0x18ec500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414d14 in worker_libevent (arg=0x18c4500) at daemon/thread.c:301
#4 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 4 (Thread 2353):
#0 0x00007f5255d0b633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5256ad9f36 in epoll_dispatch (base=0x18ecc80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f5256ac5394 in event_base_loop (base=0x18ecc80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414d14 in worker_libevent (arg=0x18c46f0) at daemon/thread.c:301
#4 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 3 (Thread 2341):
#0 0x00007f5255d0b633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5256ad9f36 in epoll_dispatch (base=0x18ec000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f5256ac5394 in event_base_loop (base=0x18ec000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004081de in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7574

Thread 2 (Thread 2366):
#0 0x00007f5255fc5f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f525179d64a in wait (this=0x6352300, d=...) at syncobject.hh:42
#2 IdleTask::run (this=0x6352300, d=...) at dispatcher.cc:286
#3 0x00007f52517a0981 in Dispatcher::run (this=0x1908700) at dispatcher.cc:123
#4 0x00007f52517a10db in launch_dispatcher_thread (arg=0x1908754) at dispatcher.cc:28
#5 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 1 (Thread 2364):
#0 0x00007f5255c55d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5255c59ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f5255c4e7c5 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f52517f3a46 in TapProducer::completeBGFetchJob (this=0x704d400, itm=0x1e35948c0, implicitEnqueue=false) at tapconn
ection.cc:1019
#4 0x00007f52517ff365 in TapBGFetchCallback::callback(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/me
mcached/ep.so
#5 0x00007f52517a19ef in Task::run(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#6 0x00007f52517a0981 in Dispatcher::run (this=0x19088c0) at dispatcher.cc:123
#7 0x00007f52517a10db in launch_dispatcher_thread (arg=0x925) at dispatcher.cc:28
#8 0x00007f5255fc0d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007f5255d0afdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 couchbase couchbase 31 2013-02-26 03:37 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached
_utilities.so.0.0.0
4553828cf520e017ee8b4a3cd897a652 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 couchbase couchbase 21 2013-02-26 03:37 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
305ae4a18396e003f5c1666998614a78 /opt/couchbase/lib/libevent-2.0.so.5
/lib/x86_64-linux-gnu/libdl.so.2:
lrwxrwxrwx 1 root root 13 2012-10-28 04:20 /lib/x86_64-linux-gnu/libdl.so.2 -> libdl-2.13.so
a2e1adba969c272af80761ad2c0ba545 /lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libm.so.6:
lrwxrwxrwx 1 root root 12 2012-10-28 04:20 /lib/x86_64-linux-gnu/libm.so.6 -> libm-2.13.so
6e8f06c6627267ae8c58104f24af3bf8 /lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/librt.so.1:
lrwxrwxrwx 1 root root 13 2012-10-28 04:20 /lib/x86_64-linux-gnu/librt.so.1 -> librt-2.13.so
0ed50675d061ec9c02ba8f9626a7c95a /lib/x86_64-linux-gnu/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.4:
lrwxrwxrwx 1 couchbase couchbase 28 2013-02-26 03:37 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.
1.0
046be77e59d0e9b3002bf67a4fed1a36 /opt/couchbase/lib/libtcmalloc_minimal.so.4
/lib/x86_64-linux-gnu/libpthread.so.0:
lrwxrwxrwx 1 root root 18 2012-10-28 04:20 /lib/x86_64-linux-gnu/libpthread.so.0 -> libpthread-2.13.so
b5c21e1fe93476e636aad2a492550c2d /lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libc.so.6:
lrwxrwxrwx 1 root root 12 2012-10-28 04:20 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.13.so
12ba94490e631fdefe06a0ead3ffb1a9 /lib/x86_64-linux-gnu/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 27 2012-10-28 04:20 /lib64/ld-linux-x86-64.so.2 -> x86_64-linux-gnu/ld-2.13.so
1f5c90f22988647820774a203615763a /lib64/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 2012-10-28 04:19 /usr/lib/x86_64-linux-gnu/libstdc+.so.6 -> libstdc+.so.6.0.14
f5548e8c4d76adfac9549a434efe6ed2 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
/lib/x86_64-linux-gnu/libgcc_s.so.1:
rw-rr- 1 root root 88384 2011-04-18 14:22 /lib/x86_64-linux-gnu/libgcc_s.so.1
a96f49d2fd4097501b06d176a16e43c5 /lib/x86_64-linux-gnu/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
rw-rr- 1 couchbase couchbase 62216 2012-06-27 00:36 /opt/couchbase/lib/memcached/stdin_term_handler.so
023c0a6b979a7760c9ec0fd1a023be31 /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 couchbase couchbase 22 2013-02-26 03:37 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
b7d33a67068b8912adb5baba9b63ee77 /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 couchbase couchbase 11 2013-02-26 03:37 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
56b079fee5f2c3d812ea67bd5a96da02 /opt/couchbase/lib/memcached/ep.so