Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
1.8.1
-
Security Level: Public
-
None
-
centos 5.x 64bit
Description
Larger cluster dgm test.
Install couchbase server 1.8.1-840 on 10 centos 5.x servers 64bit
Create a 10 nodes cluster with 6.5 GB default bucket.
Load 30 million items with 256 bytes/item and 10 million items with 512 bytes/item
Cluster goes in to dgm with 78.8 % resident ratio
Mutate 40 million items to create cache miss and disk fetch. Maintain the load around 15K to 17K ops.
Add 2 nodes (54 and 55) at the same time and rebalance.
During rebalance, reboot one node (node 61). When this node is up, memcached on this node constantly crashed
Control connection to memcached on 'ns_1@10.3.2.61' disconnected: {{badmatch,
{error,
closed}},
[
,
{mc_client_binary, stats,4},
{ns_memcached, do_handle_call, 3},
{ns_memcached, handle_call, 3},
{gen_server, handle_msg, 5},
{proc_lib, init_p_do_apply, 3}]}
Port server memcached on node 'ns_1@10.3.2.61' exited with status 134. Restarting. Messages: Extension support isn't implemented in this version of bucket_engine
Unknown error code: 13 (repeated 5 times)
Log from core dump:
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff6309f000
/tmp/core.memcached.4115.log contains initial debug information to try to identify
the problem. The log may not contain all information we may need,
so it is recommended that you preserve the corefile and the binary
for a more deep analysis. If the program use any shared libraries
(such as memcached engines) you should also preserve them.
Please send /tmp/core.memcached.4115.log to support@couchbase.com
[root@centos-57-sr4-003 bin]# more /tmp/core.memcached.4115.log
Basic crash dump analysis of /tmp/core.memcached.4115.
Please send the file to support@couchbase.com
--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 1608023 May 25 16:35 /opt/couchbase/./bin/memcached
rw------ 1 couchbase couchbase 857575424 May 29 00:41 /tmp/core.memcached.4115
4e78e59b416d2416df6af446cab89d3d /opt/couchbase/./bin/memcached
76f213dcb5902a39975f44bfa7a2f219 /tmp/core.memcached.4115
memcached 1.4.4_488_g71871b3
memcached 1.4.4_488_g71871b3
VERSION 1.4.4_488_g71871b3
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 4120]
[New Thread 4132]
[New Thread 4131]
[New Thread 4130]
[New Thread 4129]
[New Thread 4125]
[New Thread 4122]
[New Thread 4121]
[New Thread 4119]
[New Thread 4118]
[New Thread 4117]
[New Thread 4116]
[New Thread 4115]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff6309f000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00002ac2ed2ac265 in raise () from /lib64/libc.so.6
Thread 13 (Thread 0x2ac2edae4240 (LWP 4115)):
#0 0x00002ac2ed3508a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ac2ec562be8 in epoll_dispatch (base=0x2f12000, tv=<value optimized out>) at epoll.c:404
#2 0x00002ac2ec550d5a in event_base_loop (base=0x2f12000, flags=<value optimized out>) at event.c:1558
#3 0x0000000000408cd0 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7504
Thread 12 (Thread 4116):
#0 0x00002ac2ed34278b in read () from /lib64/libc.so.6
#1 0x00002ac2ed2e8d57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00002ac2ed2e971e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00002ac2ed2e4fdb in getc () from /lib64/libc.so.6
#4 0x00002ac2edae5875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19
#5 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#6 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 11 (Thread 4117):
#0 0x00002ac2ed316541 in nanosleep () from /lib64/libc.so.6
#1 0x00002ac2ed316364 in sleep () from /lib64/libc.so.6
#2 0x0000000000415c88 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
#3 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#4 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 10 (Thread 4118):
#0 0x00002ac2ed3508a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ac2ec562be8 in epoll_dispatch (base=0x2f12500, tv=<value optimized out>) at epoll.c:404
#2 0x00002ac2ec550d5a in event_base_loop (base=0x2f12500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000413704 in worker_libevent (arg=0x2eea500) at daemon/thread.c:301
#4 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#5 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 9 (Thread 4119):
#0 0x00002ac2ed3508a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ac2ec562be8 in epoll_dispatch (base=0x2f12280, tv=<value optimized out>) at epoll.c:404
#2 0x00002ac2ec550d5a in event_base_loop (base=0x2f12280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000413704 in worker_libevent (arg=0x2eea5f8) at daemon/thread.c:301
#4 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#5 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 8 (Thread 4121):
#0 0x00002ac2ed3508a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ac2ec562be8 in epoll_dispatch (base=0x2f12a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002ac2ec550d5a in event_base_loop (base=0x2f12a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000413704 in worker_libevent (arg=0x2eea7e8) at daemon/thread.c:301
#4 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#5 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 7 (Thread 4122):
#0 0x00002ac2ed3508a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ac2ec562be8 in epoll_dispatch (base=0x2f12780, tv=<value optimized out>) at epoll.c:404
#2 0x00002ac2ec550d5a in event_base_loop (base=0x2f12780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000413704 in worker_libevent (arg=0x2eea8e0) at daemon/thread.c:301
#4 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#5 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 6 (Thread 4125):
#0 0x00002ac2ed316541 in nanosleep () from /lib64/libc.so.6
#1 0x00002ac2ed349ed4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaad51818 in updateStatsThread (arg=<value optimized out>) at memory_tracker.cc:31
#3 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#4 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 5 (Thread 4129):
#0 0x00002ac2ed06bee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaad07d3a in wait (this=0x2f2c380) at syncobject.hh:31
#2 Dispatcher::run (this=0x2f2c380) at dispatcher.cc:89
#3 0x00002aaaaad087ab in launch_dispatcher_thread (arg=0x2f2c3d4) at dispatcher.cc:28
#4 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#5 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 4 (Thread 4130):
#0 0x00002ac2ed06e97b in read () from /lib64/libpthread.so.0
#1 0x00002aaaaad8a9af in seekAndRead (id=0x417a178, pBuf=0x1046de50, amt=1024, offset=381067264) at embedded/sqlite3.c:25247
#2 unixRead (id=0x417a178, pBuf=0x1046de50, amt=1024, offset=381067264) at embedded/sqlite3.c:25281
#3 0x00002aaaaad8703d in vfsepstatRead (pFile=0x417a150, zBuf=0x1046de50, iAmt=1024, iOfst=381067264) at sqlite-vfs.c:112
#4 0x00002aaaaada0196 in sqlite3OsRead (pPg=0x1046de08) at embedded/sqlite3.c:13402
#5 readDbPage (pPg=0x1046de08) at embedded/sqlite3.c:36959
#6 0x00002aaaaada10be in sqlite3PagerAcquire (pPager=0x417a008, pgno=372137, ppPage=0x46dad2c0, noContent=0) at embedded/sqlite3.c:39100
#7 0x00002aaaaadb3761 in getAndInitPage (pCur=0x583c0c8, newPgno=273079888) at embedded/sqlite3.c:46084
#8 moveToChild (pCur=0x583c0c8, newPgno=273079888) at embedded/sqlite3.c:48599
#9 0x00002aaaaaddce7e in moveToLeftmost (pCur=0x583c0c8, pRes=0x46dad4d0) at embedded/sqlite3.c:48764
#10 sqlite3BtreeNext (pCur=0x583c0c8, pRes=0x46dad4d0) at embedded/sqlite3.c:49146
#11 0x00002aaaaae2f493 in sqlite3VdbeExec (p=0x46d1208) at embedded/sqlite3.c:63760
#12 0x00002aaaaae106ea in sqlite3Step (pStmt=0x46d1208) at embedded/sqlite3.c:57947
#13 sqlite3_step (pStmt=0x46d1208) at embedded/sqlite3.c:58011
#14 0x00002aaaaad7d735 in PreparedStatement::fetch (this=0x46b8560) at sqlite-pst.cc:96
#15 0x00002aaaaad7ba8e in StrategicSqlite3::dump (this=0x415c6c0, cb=...) at sqlite-kvstore.cc:323
#16 0x00002aaaaad12c6d in EventuallyPersistentStore::warmup (this=0x413c280, vbStateLoaded=...) at ep.cc:2055
#17 0x00002aaaaad4aef0 in Flusher::initialize (this=0x42b8090, tid=<value optimized out>) at flusher.cc:117
#18 0x00002aaaaad4b318 in Flusher::step (this=0x42b8090, d=..., tid=Traceback (most recent call last):
File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string
return '%s (count %d) %s' % (self.typename,
TypeError: int argument required
) at flusher.cc:147
#19 0x00002aaaaad4ba9e in FlusherStepper::callback (this=0x52919a0, d=..., t=<value optimized out>) at flusher.cc:23
#20 0x00002aaaaad094af in Task::run (this=<value optimized out>, d=..., t=<value optimized out>) at dispatcher.hh:139
#21 0x00002aaaaad07f06 in Dispatcher::run (this=0x2f2c540) at dispatcher.cc:123
#22 0x00002aaaaad087ab in launch_dispatcher_thread (arg=0x52) at dispatcher.cc:28
#23 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#24 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 3 (Thread 4131):
#0 0x00002ac2ed06c150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaad05c10 in wait (this=0x2f9e980, d=...) at syncobject.hh:42
#2 IdleTask::run (this=0x2f9e980, d=...) at dispatcher.cc:286
#3 0x00002aaaaad07f06 in Dispatcher::run (this=0x2f2ca80) at dispatcher.cc:123
#4 0x00002aaaaad087ab in launch_dispatcher_thread (arg=0x2f2cad4) at dispatcher.cc:28
#5 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#6 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 2 (Thread 4132):
#0 0x00002ac2ed06c150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaad26ff1 in wait (this=0x2f86000) at syncobject.hh:42
#2 wait (this=0x2f86000) at syncobject.hh:58
#3 wait (this=0x2f86000) at tapconnmap.hh:171
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x2f86000) at ep_engine.cc:3844
#5 0x00002aaaaad27153 in EvpNotifyPendingConns (arg=0x2f86000) at ep_engine.cc:1072
#6 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#7 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x43ba9940 (LWP 4120)):
#0 0x00002ac2ed2ac265 in raise () from /lib64/libc.so.6
#1 0x00002ac2ed2add10 in abort () from /lib64/libc.so.6
#2 0x0000000000409d67 in process_bin_get (c=0x2f2fb88) at daemon/memcached.c:1781
#3 0x00000000004101c3 in complete_nread (c=0x1013) at daemon/memcached.c:3369
#4 conn_nread (c=0x1013) at daemon/memcached.c:5299
#5 0x00000000004063fe in event_handler (fd=50, which=<value optimized out>, arg=0x2f2fb88) at daemon/memcached.c:5541
#6 0x00002ac2ec551109 in event_base_loop (base=0x2f12c80, flags=<value optimized out>) at event.c:1308
#7 0x0000000000413704 in worker_libevent (arg=0x2eea6f0) at daemon/thread.c:301
#8 0x00002ac2ed06773d in start_thread () from /lib64/libpthread.so.0
#9 0x00002ac2ed3504bd in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 May 25 20:10 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
403f0784777ce066f130ccb942583d7f /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 May 25 20:10 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
e16ee84c6cb55c4db57ebb496947f421 /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 12 Sep 14 2011 /lib64/libdl.so.2 -> libdl-2.5.so
47fe8c8b383dca4992313fb8edfe4a89 /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 11 Sep 14 2011 /lib64/libm.so.6 -> libm-2.5.so
70ce2bcde4aef3579e610c4b6d0e61a7 /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 12 Sep 14 2011 /lib64/librt.so.1 -> librt-2.5.so
adf7d33b39f7051854e854723f672678 /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.4:
lrwxrwxrwx 1 bin bin 28 May 25 20:10 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0
a50928f67d14254eda247fb1d7622165 /opt/couchbase/lib/libtcmalloc_minimal.so.4
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 17 Sep 14 2011 /lib64/libpthread.so.0 -> libpthread-2.5.so
06b43304a38330ba4a928d2c9584744b /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 11 Sep 14 2011 /lib64/libc.so.6 -> libc-2.5.so
40a4baf51f54d1d9b38c3134bfce4fc0 /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 9 Sep 14 2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so
8df7cc56ac89d47ec6e755ea5aba6ec3 /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 18 Sep 14 2011 /usr/lib64/libstdc+.so.6 -> libstdc+.so.6.0.8
11280e717c082632a690edc9349aea5c /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Sep 14 2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1
7dc360bdfd297bc69eb679e083037342 /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 45058 May 25 16:35 /opt/couchbase/lib/memcached/stdin_term_handler.so
4e58e368155a54a87ec6c1b4c4936bc2 /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 May 25 20:10 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
e17374f50d649e6398651bb6f6ddbbea /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 May 25 20:10 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
EP engine v1.8.0_116_g9c329ef
28a6240277a21d1b120c0b329174826d /opt/couchbase/lib/memcached/ep.so