Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0.1
-
Security Level: Public
-
OS:
Description of XDCR setup:
Description of application workload:
-
Centos 64-bit
Description
With two clusters doing XDCR replication, on a regular basis the memory utilization will grow very slowly and eventually will crash with a core dump.
asic crash dump analysis of core.29720.
Please send the file to support@couchbase.com
--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 1535008 Feb 28 21:36 /opt/couchbase/bin/memcached
30c4e0e7d6ff487a025921dd773ec30b /opt/couchbase/bin/memcached
memcached 1.4.4_601_ge6f892c
memcached 1.4.4_601_ge6f892c
VERSION 1.4.4_601_ge6f892c
rw------ 1 couchbase couchbase 2588475392 Apr 27 12:54 core.29720
645f66306da0be2d16b9e2d872f8d0f1 core.29720
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 29726]
[New Thread 29727]
[New Thread 29732]
[New Thread 29734]
[New Thread 29736]
[New Thread 29720]
[New Thread 29721]
[New Thread 29723]
[New Thread 29737]
[New Thread 29724]
[New Thread 29722]
[New Thread 29725]
[New Thread 29735]
[New Thread 29733]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0xb77ea424 in __kernel_vsyscall ()
Thread 14 (Thread 0xb360bb70 (LWP 29733)):
#0 JSON_checker_char (jc=0x581e40f0, next_char=<value optimized out>) at tools/JSON_checker.c:299
#1 0xb3ef150b in checkUTF8JSON (data=0xa074b2bc ":1364725540,\"process\":\"fqc\",\"status\":0,\"errors\":[
]}],\"part_numbers\":[\"604-3157\"],\"id\":\"DYH0100024769156\",\"_source\":\"products\",\"project\":\"X128\",\"part\":\"Housing\""..., size=908) at tools/JSON_checker.c:417
#2 0xb3ed53d4 in isJSON (this=0x5a42afc0, it=..., rev=3, cb=..., del=false) at src/couch-kvstore/couch-kvstore.cc:61
#3 CouchRequest::CouchRequest (this=0x5a42afc0, it=..., rev=3, cb=..., del=false) at src/couch-kvstore/couch-kvstore.cc:248
#4 0xb3ee2c4c in CouchKVStore::set (this=0xc501680, itm=..., cb=...) at src/couch-kvstore/couch-kvstore.cc:343
#5 0xb3e6349a in EventuallyPersistentStore::flushOneDelOrSet (this=0x9415600, qi=..., rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2420
#6 0xb3e6374a in EventuallyPersistentStore::flushOne (this=0x9415600, queue=std::queue wrapping: std::deque with 1941 elements =
#7 0xb3e67750 in EventuallyPersistentStore::flushVBQueue (this=0x9415600, vb=..., vb_queue=std::queue wrapping: std::deque with 1941 elements = {...}
, vbid=201, data_age=0) at src/ep.cc:2022
#8 0xb3e68ef7 in EventuallyPersistentStore::flushOutgoingQueue (this=0x9415600, flushQueue=0x9415798, flushPhase=@0x941a9ec, nextVbid=@0x941a9f0) at src/ep.cc:1964
#9 0xb3e9824a in Flusher::doFlush (this=0x941a960) at src/flusher.cc:245
#10 0xb3e999e0 in Flusher::step (this=0x941a960, d=..., tid=...) at src/flusher.cc:158
#11 0xb3e5b3a2 in Task::run (this=0xc4e8d20, d=..., t=...) at src/dispatcher.hh:136
#12 0xb3e59fe9 in Dispatcher::run (this=0xc511100) at src/dispatcher.cc:173
#13 0xb3e5a9d5 in launch_dispatcher_thread (arg=0xc511100) at src/dispatcher.cc:28
#14 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#15 0x43578e1e in clone () from /lib/libc.so.6
Thread 13 (Thread 0xb2609b70 (LWP 29735)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x4363f664 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0xb3e57b49 in wait (this=0xc4e8b90, d=...) at src/syncobject.hh:58
#3 IdleTask::run (this=0xc4e8b90, d=...) at src/dispatcher.cc:336
#4 0xb3e59fe9 in Dispatcher::run (this=0xc510f00) at src/dispatcher.cc:173
#5 0xb3e5a9d5 in launch_dispatcher_thread (arg=0xc510f00) at src/dispatcher.cc:28
#6 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#7 0x43578e1e in clone () from /lib/libc.so.6
Thread 12 (Thread 0xb573fb70 (LWP 29725)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x43579696 in epoll_wait () from /lib/libc.so.6
#2 0xb77c9f97 in epoll_dispatch (base=0xc500d80, tv=0x0) at epoll.c:404
#3 0xb77b6463 in event_base_loop (base=0xc500d80, flags=0) at event.c:1558
#4 0x0805cd47 in worker_libevent (arg=0x9414ed8) at daemon/thread.c:301
#5 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#6 0x43578e1e in clone () from /lib/libc.so.6
Thread 11 (Thread 0xb6f4db70 (LWP 29722)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x4363f664 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0xb6f509f5 in logger_thead_main (arg=0x9412040) at extensions/loggers/file_logger.c:368
#3 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#4 0x43578e1e in clone () from /lib/libc.so.6
Thread 10 (Thread 0xb5f40b70 (LWP 29724)):
#0 0xb7789b3a in MallocExtension_GetAllocatedSize () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#1 0xb3e9e4d9 in DeleteHook (ptr=0x5c31c6a0) at src/memory_tracker.cc:56
#2 0xb77862c0 in MallocHook::InvokeDeleteHookSlow(void const*) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#3 0xb777a0d2 in MallocHook::InvokeDeleteHook(void const*) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#4 0xb778c2f4 in tc_delete () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#5 0xb3eaf055 in ~SingleThreadedRCPtr (this=0x5c6aaa00, shouldPause=@0xb5f3df9f) at src/atomic.hh:412
#6 TapProducer::nextFgFetched_UNLOCKED (this=0x5c6aaa00, shouldPause=@0xb5f3df9f) at src/tapconnection.cc:1539
#7 0xb3eb23b0 in TapProducer::getNextItem (this=0x5c6aaa00, c=0xc49e000, vbucket=0xb5f40228, ret=@0xb5f3e048, referenced=@0xb5f3e04f) at src/tapconnection.cc:1850
#8 0xb3e94d38 in EventuallyPersistentEngine::doWalkTapQueue (this=0xc54e000, cookie=0xc49e000, itm=0xb5f40224, es=0xb5f40220, nes=0xb5f4022c, ttl=0xb5f4022f "\377", flags=0xb5f4022a, seqno=0xb5f4021c, vbucket=0xb5f40228, connection=0x5c6aaa00, retry=@0xb5f3e0bf) at src/ep_engine.cc:1659
#9 0xb3e84bc3 in EventuallyPersistentEngine::walkTapQueue (this=0xc54e000, cookie=0xc49e000, itm=0xb5f40224, es=0xb5f40220, nes=0xb5f4022c, ttl=0xb5f4022f "\377", flags=0xb5f4022a, seqno=0xb5f4021c, vbucket=0xb5f40228) at src/ep_engine.cc:1734
#10 0xb3e84d56 in EvpTapIterator (handle=0xc54e000, cookie=0xc49e000, itm=0xb5f40224, es=0xb5f40220, nes=0xb5f4022c, ttl=0xb5f4022f "\377", flags=0xb5f4022a, seqno=0xb5f4021c, vbucket=0xb5f40228) at src/ep_engine.cc:1054
#11 0xb674702f in bucket_tap_iterator_shim (handle=0xb674c220, cookie=0xc49e000, itm=0xb5f40224, engine_specific=0xb5f40220, nengine_specific=0xb5f4022c, ttl=0xb5f4022f "\377", flags=0xb5f4022a, seqno=0xb5f4021c, vbucket=0xb5f40228) at bucket_engine.c:1971
#12 0x08052de0 in ship_tap_log (c=0xc49e000) at daemon/memcached.c:2614
#13 0x0805c0fe in conn_ship_log (c=0xc49e000) at daemon/memcached.c:5523
#14 0x0804d885 in event_handler (fd=67, which=2, arg=0xc49e000) at daemon/memcached.c:5936
#15 0xb77b6568 in event_process_active_single_queue (base=0xc500600, flags=0) at event.c:1308
#16 event_process_active (base=0xc500600, flags=0) at event.c:1375
#17 event_base_loop (base=0xc500600, flags=0) at event.c:1572
#18 0x0805cd47 in worker_libevent (arg=0x9414e4c) at daemon/thread.c:301
#19 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#20 0x43578e1e in clone () from /lib/libc.so.6
Thread 9 (Thread 0xb1607b70 (LWP 29737)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x4363f664 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0xb3e79a0e in wait (this=0xc54e000) at src/syncobject.hh:58
#3 wait (this=0xc54e000) at src/syncobject.hh:74
#4 wait (this=0xc54e000) at src/tapconnmap.hh:169
#5 EventuallyPersistentEngine::notifyPendingConnections (this=0xc54e000) at src/ep_engine.cc:3423
#6 0xb3e79b22 in EvpNotifyPendingConns (arg=0xc54e000) at src/ep_engine.cc:1145
#7 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#8 0x43578e1e in clone () from /lib/libc.so.6
Thread 8 (Thread 0xb6741b70 (LWP 29723)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x43579696 in epoll_wait () from /lib/libc.so.6
#2 0xb77c9f97 in epoll_dispatch (base=0xc500180, tv=0x0) at epoll.c:404
#3 0xb77b6463 in event_base_loop (base=0xc500180, flags=0) at event.c:1558
#4 0x0805cd47 in worker_libevent (arg=0x9414dc0) at daemon/thread.c:301
#5 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#6 0x43578e1e in clone () from /lib/libc.so.6
Thread 7 (Thread 0xb7761b70 (LWP 29721)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x43568aeb in read () from /lib/libc.so.6
#2 0x4350461b in _IO_new_file_underflow () from /lib/libc.so.6
#3 0x4350633b in _IO_default_uflow_internal () from /lib/libc.so.6
#4 0x4350794a in __uflow () from /lib/libc.so.6
#5 0x434fa33c in _IO_getline_info_internal () from /lib/libc.so.6
#6 0x434fa281 in _IO_getline_internal () from /lib/libc.so.6
#7 0x434f91ba in fgets () from /lib/libc.so.6
#8 0xb77a67b7 in check_stdin_thread (arg=0x804a790) at extensions/daemon/stdin_check.c:37
#9 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#10 0x43578e1e in clone () from /lib/libc.so.6
Thread 6 (Thread 0xb7763900 (LWP 29720)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x43579696 in epoll_wait () from /lib/libc.so.6
#2 0xb77c9f97 in epoll_dispatch (base=0xc500000, tv=0xbfd5d254) at epoll.c:404
#3 0xb77b6463 in event_base_loop (base=0xc500000, flags=0) at event.c:1558
#4 0x08051671 in main (argc=19, argv=0xbfd5e8a4) at daemon/memcached.c:7918
Thread 5 (Thread 0xb1e08b70 (LWP 29736)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x4363f664 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0xb3e57b49 in wait (this=0xc4e8b40, d=...) at src/syncobject.hh:58
#3 IdleTask::run (this=0xc4e8b40, d=...) at src/dispatcher.cc:336
#4 0xb3e59fe9 in Dispatcher::run (this=0xc511500) at src/dispatcher.cc:173
#5 0xb3e5a9d5 in launch_dispatcher_thread (arg=0xc511500) at src/dispatcher.cc:28
#6 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#7 0x43578e1e in clone () from /lib/libc.so.6
Thread 4 (Thread 0xb2e0ab70 (LWP 29734)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x4363f664 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0xb3e57b49 in wait (this=0xc4e8a50, d=...) at src/syncobject.hh:58
#3 IdleTask::run (this=0xc4e8a50, d=...) at src/dispatcher.cc:336
#4 0xb3e59fe9 in Dispatcher::run (this=0xc511000) at src/dispatcher.cc:173
#5 0xb3e5a9d5 in launch_dispatcher_thread (arg=0xc511000) at src/dispatcher.cc:28
#6 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#7 0x43578e1e in clone () from /lib/libc.so.6
Thread 3 (Thread 0xb3e20b70 (LWP 29732)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x43537be6 in nanosleep () from /lib/libc.so.6
#2 0x43571d2c in usleep () from /lib/libc.so.6
#3 0xb3e9e7ff in updateStatsThread (arg=0x940c180) at src/memory_tracker.cc:31
#4 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#5 0x43578e1e in clone () from /lib/libc.so.6
Thread 2 (Thread 0xb473db70 (LWP 29727)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x43579696 in epoll_wait () from /lib/libc.so.6
#2 0xb77c9f97 in epoll_dispatch (base=0xc501500, tv=0x0) at epoll.c:404
#3 0xb77b6463 in event_base_loop (base=0xc501500, flags=0) at event.c:1558
#4 0x0805cd47 in worker_libevent (arg=0x9414ff0) at daemon/thread.c:301
#5 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#6 0x43578e1e in clone () from /lib/libc.so.6
Thread 1 (Thread 0xb4f3eb70 (LWP 29726)):
#0 0xb77ea424 in __kernel_vsyscall ()
#1 0x434c4b01 in raise () from /lib/libc.so.6
#2 0x434c63da in abort () from /lib/libc.so.6
#3 0x434bdddb in __assert_fail_base () from /lib/libc.so.6
#4 0x434bde96 in __assert_fail () from /lib/libc.so.6
#5 0xb3ef2116 in ObjectRegistry::onCreateBlob (blob=0x59fa5700) at src/objectregistry.cc:57
#6 0xb3e7c6b4 in Blob (this=0xc54e000, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/item.hh:116
#7 New (this=0xc54e000, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/item.hh:60
#8 setData (this=0xc54e000, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/item.hh:348
#9 Item (this=0xc54e000, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/item.hh:163
#10 EventuallyPersistentEngine::setWithMeta (this=0xc54e000, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/ep_engine.cc:3876
#11 0xb3e867d6 in processUnknownCommand (h=<value optimized out>, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/ep_engine.cc:949
#12 0xb3e87af1 in EvpUnknownCommand (handle=0xc54e000, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at src/ep_engine.cc:1013
#13 0xb6746901 in bucket_unknown_command (handle=0xb674c220, cookie=0xc49e3c0, request=0x577f4000, response=0x804f240 <binary_response_handler>) at bucket_engine.c:2473
#14 0x08059e57 in process_bin_unknown_packet (c=0xc49e3c0) at daemon/memcached.c:2876
#15 process_bin_packet (c=0xc49e3c0) at daemon/memcached.c:3164
#16 complete_nread_binary (c=0xc49e3c0) at daemon/memcached.c:3738
#17 complete_nread (c=0xc49e3c0) at daemon/memcached.c:3820
#18 conn_nread (c=0xc49e3c0) at daemon/memcached.c:5673
#19 0x0804d885 in event_handler (fd=56, which=2, arg=0xc49e3c0) at daemon/memcached.c:5936
#20 0xb77b6568 in event_process_active_single_queue (base=0xc500a80, flags=0) at event.c:1308
#21 event_process_active (base=0xc500a80, flags=0) at event.c:1375
#22 event_base_loop (base=0xc500a80, flags=0) at event.c:1572
#23 0x0805cd47 in worker_libevent (arg=0x9414f64) at daemon/thread.c:301
#24 0x4363ba49 in start_thread () from /lib/libpthread.so.0
#25 0x43578e1e in clone () from /lib/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
/opt/couchbase/lib/libevent-2.0.so.5:
/lib/libdl.so.2:
/lib/libm.so.6:
/lib/librt.so.1:
/opt/couchbase/lib/libtcmalloc_minimal.so.4:
/lib/libpthread.so.0:
/lib/libc.so.6:
/lib/ld-linux.so.2:
/usr/lib/libstdc++.so.6:
/lib/libgcc_s.so.1:
/opt/couchbase/lib/memcached/stdin_term_handler.so:
/opt/couchbase/lib/memcached/file_logger.so:
/opt/couchbase/lib/memcached/bucket_engine.so:
/opt/couchbase/lib/memcached/ep.so:
/opt/couchbase/lib/libcouchstore.so.1:
/opt/couchbase/lib/libsnappy.so.1:
/lib/libnss_files.so.2:
Attachments
For Gerrit Dashboard: MB-8166 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
26269,2 | MB-8166 Check an item value size in setWithMeta operation. | 2.0.2 | ep-engine | Status: MERGED | +2 | +1 |