Details
-
Task
-
Resolution: Duplicate
-
Major
-
None
-
2.0, 2.0.1
-
Security Level: Public
-
None
Description
the same test as in MB-7571 -> MB-7771
3 nodes with 2.0.0, rebalance out 2 nodes, in 2 nodes 2.0.1-163
3 buckets *1.5 M items, 1 ddoc*1 view
resident ratio ~50 %
Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'ns_1@172.27.33.2',
{'EXIT',
{{{{unexpected_reason,killed},
[
,
{tap_replication_manager, change_vbucket_filter,4},
{tap_replication_manager, '-do_set_incoming_replication_map/3-lc$^5/1-5-', 2},
{tap_replication_manager, do_set_incoming_replication_map,3},
{tap_replication_manager,handle_call,3},
{gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]},{gen_server,call,
['tap_replication_manager-standard', {change_vbucket_replication,621, 'ns_1@172.27.33.5'},
infinity]}},
{gen_server,call,
[{'janitor_agent-standard', 'ns_1@172.27.33.2'},
{if_rebalance,<0.3140.0>,
{update_vbucket_state,621,replica,
undefined,'ns_1@172.27.33.5'}},
infinity]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,on_move_done,2},{gen_server,handle_msg,5}
,
{proc_lib,init_p_do_apply,3}]}
ns_orchestrator002 ns_1@172.27.33.5 19:49:42 - Wed Feb 20, 2013
on orchestrator 172.27.33.5( 2.0.0) :
[root@kiwi-r101 tmp]# free
total used free shared buffers cached
Mem: 4194304 4176520 17784 0 10656 573816
-/+ buffers/cache: 3592048 602256
Swap: 4843588 2303804 2539784
[root@kiwi-r101 tmp]# top
top - 20:53:55 up 64 days, 17:20, 1 user, load average: 13.08, 13.43, 14.40
Tasks: 135 total, 2 running, 133 sleeping, 0 stopped, 0 zombie
Cpu(s): 77.8%us, 4.1%sy, 0.0%ni, 2.7%id, 15.0%wa, 0.0%hi, 0.4%si, 0.1%st
Mem: 4194304k total, 4171220k used, 23084k free, 10756k buffers
Swap: 4843588k total, 2303804k used, 2539784k free, 574300k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17029 couchbas 15 0 3081m 2.3g 2376 S 305.7 58.5 218:28.11 memcached
16859 couchbas 25 0 2775m 661m 3656 S 21.3 16.2 50:39.96 beam.smp
17052 couchbas 18 0 6012 460 400 S 0.7 0.0 0:15.98 sigar_port
1 root 15 0 10364 564 536 S 0.0 0.0 0:00.54 init
I see almost no RAM but memcached processes are eating up 300% cpu
I found that was bug in 2.0.0 CBSE-340 that will be fixed in 2.0.1
Alk, maybe it is due to the fall in the rebalance during online update when server is loaded as in bugs MB-7571, MB-7771
please close this bug, if all this is due to the high server load
gdb backtrace of the memcached process on 2.0.0 master
[root@kiwi-r101 tmp]# ps -ef| grep memcac
root 1814 572 0 20:59 pts/1 00:00:00 grep memcac
101 17029 16859 82 16:13 ? 03:55:17 /opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler.so -X /opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/opt/couchbase/var/lib/couchbase/logs/memcached.log -l 0.0.0.0:11210,0.0.0.0:11209:1000 -p 11210 -E /opt/couchbase/lib/memcached/bucket_engine.so -B binary -r -c 10000 -e admin=_admin;default_bucket_name=default;auto_create=false
[root@kiwi-r101 tmp]# gdb --pid 17029 -ex 'thread apply all bt' < /dev/null
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 17029
Reading symbols from /opt/couchbase/bin/memcached...done.
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x4ea4a940 (LWP 17856)]
[New Thread 0x4e049940 (LWP 17855)]
[New Thread 0x4d648940 (LWP 17854)]
[New Thread 0x4cc47940 (LWP 17853)]
[New Thread 0x4c246940 (LWP 17852)]
[New Thread 0x4b845940 (LWP 17434)]
[New Thread 0x4ae44940 (LWP 17433)]
[New Thread 0x4a443940 (LWP 17432)]
[New Thread 0x49a42940 (LWP 17431)]
[New Thread 0x49041940 (LWP 17430)]
[New Thread 0x48640940 (LWP 17427)]
[New Thread 0x47c3f940 (LWP 17426)]
[New Thread 0x4723e940 (LWP 17425)]
[New Thread 0x4683d940 (LWP 17424)]
[New Thread 0x45e3c940 (LWP 17423)]
[New Thread 0x4543b940 (LWP 17422)]
[New Thread 0x44a3a940 (LWP 17057)]
[New Thread 0x44039940 (LWP 17056)]
[New Thread 0x43638940 (LWP 17055)]
[New Thread 0x42c37940 (LWP 17054)]
[New Thread 0x42236940 (LWP 17053)]
[New Thread 0x41835940 (LWP 17038)]
[New Thread 0x40e34940 (LWP 17037)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff4e961000
0x00000033a3ad4008 in epoll_wait () from /lib64/libc.so.6
Thread 24 (Thread 0x40e34940 (LWP 17037)):
#0 0x00000033a3ac5e1b in read () from /lib64/libc.so.6
#1 0x00000033a3a6bad7 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00000033a3a6c49e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00000033a3a61584 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00000033a3a60429 in fgets () from /lib64/libc.so.6
#5 0x00002b633ae56939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#7 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x41835940 (LWP 17038)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x8778040) at extensions/loggers/file_logger.c:368
#2 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#3 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x42236940 (LWP 17053)):
#0 length (this=<value optimized out>) at src/item.hh:78
#1 StoredValue::size (this=<value optimized out>) at src/stored-value.hh:514
#2 0x00002aaaaaf096d4 in HashTable::set(Item const&, unsigned long, bool, bool, bool) () from /opt/couchbase/lib/memcached/ep.so
#3 0x00002aaaaaefe976 in EventuallyPersistentStore::setWithMeta (this=0xcfb1b00, itm=..., cas=0, cookie=<value optimized out>, force=<value optimized out>, allowExisting=true, trackReference=true)
at src/ep.cc:1383
#4 0x00002aaaaaf16e10 in EventuallyPersistentEngine::tapNotify (this=0xcfb4900, cookie=0xcf5f340, engine_specific=0x2aab0e626028, nengine=1, tap_flags=<value optimized out>, tap_event=TAP_MUTATION,
tap_seqno=2992459, key=0x2aab0e626031, nkey=21, flags=0, exptime=0, cas=5585144921030929, data=0x2aab0e626046, ndata=1248, vbucket=452) at src/ep_engine.cc:2065
#5 0x00002aaaaaf17638 in EvpTapNotify (handle=0xcfb4900, cookie=0xcf5f340, engine_specific=0x2aab0e626028, nengine=9, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=2992459,
key=0x2aab0e626031, nkey=21, flags=0, exptime=0, cas=5585144921030929, data=0x2aab0e626046, ndata=1248, vbucket=452) at src/ep_engine.cc:1040
#6 0x00002aaaaacc6714 in bucket_tap_notify (handle=<value optimized out>, cookie=0xcf5f340, engine_specific=0x2aab0e626028, nengine=9, ttl=254 '\376', tap_flags=<value optimized out>,
tap_event=TAP_MUTATION, tap_seqno=2992459, key=0x2aab0e626031, nkey=21, flags=0, exptime=0, cas=5585144921030929, data=0x2aab0e626046, ndata=1248, vbucket=452) at bucket_engine.c:1942
#7 0x000000000040cb02 in process_bin_tap_packet (event=TAP_MUTATION, c=0xcf5f340) at daemon/memcached.c:3031
#8 0x0000000000411851 in process_bin_packet (c=0xcf5f340) at daemon/memcached.c:3117
#9 complete_nread_binary (c=0xcf5f340) at daemon/memcached.c:3738
#10 complete_nread (c=0xcf5f340) at daemon/memcached.c:3820
#11 conn_nread (c=0xcf5f340) at daemon/memcached.c:5673
#12 0x0000000000405e55 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0xcf5f340) at daemon/memcached.c:5936
#13 0x00002b633a9c5f3c in event_process_active_single_queue (base=0xcfb8500, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0xcfb8500, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0xcfb8500, flags=<value optimized out>) at event.c:1572
#16 0x0000000000414504 in worker_libevent (arg=0x877b900) at daemon/thread.c:301
#17 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#18 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x42c37940 (LWP 17054)):
#0 0x00000033a3ad4008 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b633a9da576 in epoll_dispatch (base=0xcfb8280, tv=<value optimized out>) at epoll.c:404
--Type <return> to continue, or q <return> to quit--#2 0x00002b633a9c5e44 in event_base_loop (base=0xcfb8280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x877b9f8) at daemon/thread.c:301
#4 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#5 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x43638940 (LWP 17055)):
#0 0x00000033a3ad4008 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b633a9da576 in epoll_dispatch (base=0xcfb8c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002b633a9c5e44 in event_base_loop (base=0xcfb8c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x877baf0) at daemon/thread.c:301
#4 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#5 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x44039940 (LWP 17056)):
#0 0x00000033a3ad4008 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b633a9da576 in epoll_dispatch (base=0xcfb8a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002b633a9c5e44 in event_base_loop (base=0xcfb8a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x877bbe8) at daemon/thread.c:301
#4 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#5 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x44a3a940 (LWP 17057)):
#0 0x00000033a3ad4008 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b633a9da576 in epoll_dispatch (base=0xcfb8780, tv=<value optimized out>) at epoll.c:404
#2 0x00002b633a9c5e44 in event_base_loop (base=0xcfb8780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x877bce0) at daemon/thread.c:301
#4 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#5 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x4543b940 (LWP 17422)):
#0 0x00000033a3a99be1 in nanosleep () from /lib64/libc.so.6
#1 0x00000033a3acd564 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf314d5 in updateStatsThread (arg=0x87784c0) at src/memory_tracker.cc:31
#3 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#4 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x45e3c940 (LWP 17423)):
#0 0x00000033a420e508 in pread64 () from /lib64/libpthread.so.0
#1 0x00002aaaab1d0118 in couch_pread (handle=<value optimized out>, buf=0x2aab214bc831, nbyte=8192, offset=401408) at src/os.c:38
#2 0x00002aaaaaf738fd in cfs_pread (h=0x2aaaff8872e0, buf=0x2aab214bc831, sz=8192, off=401408) at src/couch-kvstore/couch-fs-stats.cc:68
#3 0x00002aaaab1ce82b in load_buffer_from (handle=0x2aaac8fdfbf0, buf=<value optimized out>, nbyte=8, offset=<value optimized out>) at src/iobuffer.c:153
#4 buffered_pread (handle=0x2aaac8fdfbf0, buf=<value optimized out>, nbyte=8, offset=<value optimized out>) at src/iobuffer.c:292
#5 0x00002aaaab1ccd8c in read_skipping_prefixes (db=0xcfbeb60, pos=407706, ret_ptr=0x45e3b1a8, header=0) at src/couch_file_read.c:27
#6 pread_bin_internal (db=0xcfbeb60, pos=407706, ret_ptr=0x45e3b1a8, header=0) at src/couch_file_read.c:53
#7 0x00002aaaab1ccf97 in pread_compressed (db=<value optimized out>, pos=<value optimized out>, ret_ptr=0x45e3b248) at src/couch_file_read.c:92
#8 0x00002aaaab1c896a in modify_node (rq=0x45e3b480, nptr=0x2aab1ef03dd8, start=124, end=126, dst=0x2aab1ef03010) at src/btree_modify.c:317
#9 0x00002aaaab1c8b5b in modify_node (rq=0x45e3b480, nptr=0x2aab1ef02fe0, start=124, end=148, dst=0x2aab1eefc0c8) at src/btree_modify.c:458
#10 0x00002aaaab1c8b5b in modify_node (rq=0x45e3b480, nptr=0x2aab21d82740, start=112, end=514, dst=0x2aab1eefc010) at src/btree_modify.c:458
#11 0x00002aaaab1c9317 in modify_btree (rq=0x45e3b480, root=0x2aab21d82740, errcode=0x45e3b58c) at src/btree_modify.c:607
#12 0x00002aaaab1cc0e5 in update_indexes (db=0xcfbeb60, seqs=0x2aab2299e010, seqvals=0x2aab229a0030, ids=<value optimized out>, idvals=0x2aab229a1040, numdocs=257) at src/couch_save.c:204
#13 0x00002aaaab1ccbc0 in couchstore_save_documents (db=0xcfbeb60, docs=0x19fd32f8, infos=<value optimized out>, numdocs=257, options=1) at src/couch_save.c:359
#14 0x00002aaaaaf6eb37 in CouchKVStore::saveDocs (this=0xd020000, vbid=346, rev=2, docs=0xcfb5200, docinfos=0xcfb6d00, docCount=257) at src/couch-kvstore/couch-kvstore.cc:1556
#15 0x00002aaaaaf6f241 in CouchKVStore::commit2couchstore (this=0xd020000) at src/couch-kvstore/couch-kvstore.cc:1491
#16 0x00002aaaaaf6f70c in queueItem (this=0xd020000, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:1623
#17 CouchKVStore::del (this=0xd020000, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:510
--Type <return> to continue, or q <return> to quit--#18 0x00002aaaaaefcfb1 in EventuallyPersistentStore::flushOneDelOrSet (this=0xcfb0480, qi=..., rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2438
#19 0x00002aaaaaefd3db in EventuallyPersistentStore::flushOne (this=0xcfb0480, queue=<value optimized out>, rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2460
#20 0x00002aaaaaf00ed5 in EventuallyPersistentStore::flushVBQueue (this=0xcfb0480, vb=..., vb_queue=std::queue wrapping: std::deque with 1256 elements =
#21 0x00002aaaaaf0212c in EventuallyPersistentStore::flushOutgoingQueue (this=0xcfb0480, flushQueue=0xcfb0748, flushPhase=@0xcfae570, nextVbid=@0xcfae578) at src/ep.cc:1956
#22 0x00002aaaaaf2b55c in Flusher::doFlush (this=0xcfae480) at src/flusher.cc:245
#23 0x00002aaaaaf2c395 in Flusher::step (this=0xcfae480, d=..., tid=...) at src/flusher.cc:158
#24 0x00002aaaaaef459a in Dispatcher::run (this=0xcff4c40) at src/dispatcher.cc:173
#25 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xcff4c40) at src/dispatcher.cc:28
#26 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#27 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x4683d940 (LWP 17424)):
#0 SpinLock::release (this=0xd02df50) at src/atomic.cc:45
#1 0x00002aaaaaf52b32 in unlock (this=<value optimized out>, id=<value optimized out>) at src/atomic.hh:265
#2 ~SpinLockHolder (this=<value optimized out>, id=<value optimized out>) at src/atomic.hh:255
#3 gimme (this=<value optimized out>, id=<value optimized out>) at src/atomic.hh:377
#4 RCPtr (this=<value optimized out>, id=<value optimized out>) at src/atomic.hh:311
#5 VBucketMap::getBucket (this=<value optimized out>, id=<value optimized out>) at src/vbucketmap.cc:32
#6 0x00002aaaaaee7273 in BgFetcher::run (this=0xd0341b0, tid=<value optimized out>) at src/bgfetcher.cc:107
#7 0x00002aaaaaef459a in Dispatcher::run (this=0xcff4a80) at src/dispatcher.cc:173
#8 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xcff4a80) at src/dispatcher.cc:28
#9 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#10 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x4723e940 (LWP 17425)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef1ed8 in wait (this=0xd0342d0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0xd0342d0, d=...) at src/dispatcher.cc:336
#3 0x00002aaaaaef459a in Dispatcher::run (this=0xcff5880) at src/dispatcher.cc:173
#4 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xcff5880) at src/dispatcher.cc:28
#5 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#6 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x47c3f940 (LWP 17426)):
#0 length (this=0x1cab77a0, stats=..., ht=...) at src/item.hh:78
#1 size (this=0x1cab77a0, stats=..., ht=...) at src/stored-value.hh:514
#2 StoredValue::ejectValue (this=0x1cab77a0, stats=..., ht=...) at src/stored-value.cc:31
#3 0x00002aaaaaf3070e in PagingVisitor::visit (this=0x8787c30, v=0x1cab77a0) at src/item_pager.cc:69
#4 0x00002aaaaaf367f5 in HashTable::visit (this=0xe498008, visitor=...) at src/stored-value.cc:404
#5 0x00002aaaaaef8ab2 in VBCBAdaptor::callback (this=0x2aaaffe3d7a0, d=..., t=...) at src/ep.cc:2842
#6 0x00002aaaaaef459a in Dispatcher::run (this=0xcff56c0) at src/dispatcher.cc:173
#7 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xcff56c0) at src/dispatcher.cc:28
#8 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#9 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x48640940 (LWP 17427)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf10daf in wait (this=0xcfb4000) at src/syncobject.hh:58
#2 wait (this=0xcfb4000) at src/syncobject.hh:74
#3 wait (this=0xcfb4000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0xcfb4000) at src/ep_engine.cc:3415
#5 0x00002aaaaaf10e93 in EvpNotifyPendingConns (arg=0xcfb4000) at src/ep_engine.cc:1142
#6 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#7 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
--
#0 0x00000033a420e508 in pread64 () from /lib64/libpthread.so.0
#1 0x00002aaaab1d0118 in couch_pread (handle=<value optimized out>, buf=0x28198831, nbyte=8192, offset=1851392) at src/os.c:38
#2 0x00002aaaaaf738fd in cfs_pread (h=0x2aaab56c4460, buf=0x28198831, sz=8192, off=1851392) at src/couch-kvstore/couch-fs-stats.cc:68
#3 0x00002aaaab1ce82b in load_buffer_from (handle=0x2aab0f234870, buf=<value optimized out>, nbyte=8, offset=<value optimized out>) at src/iobuffer.c:153
#4 buffered_pread (handle=0x2aab0f234870, buf=<value optimized out>, nbyte=8, offset=<value optimized out>) at src/iobuffer.c:292
#5 0x00002aaaab1ccd8c in read_skipping_prefixes (db=0xcfbed20, pos=1853888, ret_ptr=0x49040258, header=0) at src/couch_file_read.c:27
#6 pread_bin_internal (db=0xcfbed20, pos=1853888, ret_ptr=0x49040258, header=0) at src/couch_file_read.c:53
#7 0x00002aaaab1ccf97 in pread_compressed (db=<value optimized out>, pos=<value optimized out>, ret_ptr=0x490402f8) at src/couch_file_read.c:92
#8 0x00002aaaab1c896a in modify_node (rq=0x49040480, nptr=0x2aab0e770180, start=0, end=564, dst=0x2aab0e7700c8) at src/btree_modify.c:317
#9 0x00002aaaab1c8b5b in modify_node (rq=0x49040480, nptr=0x2aaae0864c40, start=0, end=2456, dst=0x2aab0e770010) at src/btree_modify.c:458
#10 0x00002aaaab1c9317 in modify_btree (rq=0x49040480, root=0x2aaae0864c40, errcode=0x4904058c) at src/btree_modify.c:607
#11 0x00002aaaab1cc0e5 in update_indexes (db=0xcfbed20, seqs=0xd268010, seqvals=0xd271990, ids=<value optimized out>, idvals=0xd276650, numdocs=1228) at src/couch_save.c:204
#12 0x00002aaaab1ccbc0 in couchstore_save_documents (db=0xcfbed20, docs=0x2aab20eb53b8, infos=<value optimized out>, numdocs=1228, options=1) at src/couch_save.c:359
#13 0x00002aaaaaf6eb37 in CouchKVStore::saveDocs (this=0xd020840, vbid=389, rev=2, docs=0x2aab20a96800, docinfos=0x2aab20a94000, docCount=1228) at src/couch-kvstore/couch-kvstore.cc:1556
#14 0x00002aaaaaf6f241 in CouchKVStore::commit2couchstore (this=0xd020840) at src/couch-kvstore/couch-kvstore.cc:1491
#15 0x00002aaaaaf6f559 in queueItem (this=0xd020840, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:1623
#16 CouchKVStore::set (this=0xd020840, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:344
#17 0x00002aaaaaefd114 in EventuallyPersistentStore::flushOneDelOrSet (this=0xcfb1b00, qi=..., rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2412
#18 0x00002aaaaaefd3db in EventuallyPersistentStore::flushOne (this=0xcfb1b00, queue=<value optimized out>, rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2460
#19 0x00002aaaaaf00ed5 in EventuallyPersistentStore::flushVBQueue (this=0xcfb1b00, vb=..., vb_queue=std::queue wrapping: std::deque with 1598 elements = {...}
, vbid=390, data_age=0) at src/ep.cc:2014
#20 0x00002aaaaaf0212c in EventuallyPersistentStore::flushOutgoingQueue (this=0xcfb1b00, flushQueue=0xcfb1dc8, flushPhase=@0xcfaed50, nextVbid=@0xcfaed58) at src/ep.cc:1956
#21 0x00002aaaaaf2b55c in Flusher::doFlush (this=0xcfaec60) at src/flusher.cc:245
#22 0x00002aaaaaf2c395 in Flusher::step (this=0xcfaec60, d=..., tid=...) at src/flusher.cc:158
#23 0x00002aaaaaef459a in Dispatcher::run (this=0xd000700) at src/dispatcher.cc:173
#24 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd000700) at src/dispatcher.cc:28
#25 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#26 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x49a42940 (LWP 17431)):
#0 0x00000033a4208d46 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00002aaaaaf324ea in Mutex::acquire (this=0x10440f00) at src/mutex.cc:80
#2 0x00002aaaaaf4f3a3 in lock (this=0x10440c00, fetches=std::tr1::unordered_map with 0 elements) at src/locks.hh:48
#3 LockHolder (this=0x10440c00, fetches=std::tr1::unordered_map with 0 elements) at src/locks.hh:26
#4 VBucket::getBGFetchItems (this=0x10440c00, fetches=std::tr1::unordered_map with 0 elements) at src/vbucket.cc:213
#5 0x00002aaaaaee729b in BgFetcher::run (this=0xd034900, tid=<value optimized out>) at src/bgfetcher.cc:109
#6 0x00002aaaaaef459a in Dispatcher::run (this=0xd000540) at src/dispatcher.cc:173
#7 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd000540) at src/dispatcher.cc:28
#8 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#9 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x4a443940 (LWP 17432)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef1ed8 in wait (this=0xd034750, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0xd034750, d=...) at src/dispatcher.cc:336
#3 0x00002aaaaaef459a in Dispatcher::run (this=0xd001340) at src/dispatcher.cc:173
#4 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd001340) at src/dispatcher.cc:28
#5 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#6 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x4ae44940 (LWP 17433)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef1ed8 in wait (this=0xd034990, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0xd034990, d=...) at src/dispatcher.cc:336
#3 0x00002aaaaaef459a in Dispatcher::run (this=0xd001180) at src/dispatcher.cc:173
--Type <return> to continue, or q <return> to quit--#4 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd001180) at src/dispatcher.cc:28
#5 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#6 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x4b845940 (LWP 17434)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf10daf in wait (this=0xcfb4900) at src/syncobject.hh:58
#2 wait (this=0xcfb4900) at src/syncobject.hh:74
#3 wait (this=0xcfb4900) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0xcfb4900) at src/ep_engine.cc:3415
#5 0x00002aaaaaf10e93 in EvpNotifyPendingConns (arg=0xcfb4900) at src/ep_engine.cc:1142
#6 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#7 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x4c246940 (LWP 17852)):
#0 0x00000033a420e508 in pread64 () from /lib64/libpthread.so.0
#1 0x00002aaaab1d0118 in couch_pread (handle=<value optimized out>, buf=0xf1a6431, nbyte=8192, offset=516096) at src/os.c:38
#2 0x00002aaaaaf738fd in cfs_pread (h=0x2aaac17fae40, buf=0xf1a6431, sz=8192, off=516096) at src/couch-kvstore/couch-fs-stats.cc:68
#3 0x00002aaaab1ce82b in load_buffer_from (handle=0x2aab1a30fdd0, buf=<value optimized out>, nbyte=8, offset=<value optimized out>) at src/iobuffer.c:153
#4 buffered_pread (handle=0x2aab1a30fdd0, buf=<value optimized out>, nbyte=8, offset=<value optimized out>) at src/iobuffer.c:292
#5 0x00002aaaab1ccd8c in read_skipping_prefixes (db=0xcfbf340, pos=520913, ret_ptr=0x4c2451a8, header=0) at src/couch_file_read.c:27
#6 pread_bin_internal (db=0xcfbf340, pos=520913, ret_ptr=0x4c2451a8, header=0) at src/couch_file_read.c:53
#7 0x00002aaaab1ccf97 in pread_compressed (db=<value optimized out>, pos=<value optimized out>, ret_ptr=0x4c245248) at src/couch_file_read.c:92
#8 0x00002aaaab1c896a in modify_node (rq=0x4c2454e0, nptr=0x2aab1f4e68bc, start=7, end=8, dst=0x2aab1f4e66a8) at src/btree_modify.c:317
#9 0x00002aaaab1c8b5b in modify_node (rq=0x4c2454e0, nptr=0x2aab1f4e6678, start=7, end=10, dst=0x2aab1f4e40c8) at src/btree_modify.c:458
#10 0x00002aaaab1c8b5b in modify_node (rq=0x4c2454e0, nptr=0x2aaaea31e3c0, start=7, end=431, dst=0x2aab1f4e4010) at src/btree_modify.c:458
#11 0x00002aaaab1c9317 in modify_btree (rq=0x4c2454e0, root=0x2aaaea31e3c0, errcode=0x4c24558c) at src/btree_modify.c:607
#12 0x00002aaaab1cc229 in update_indexes (db=0xcfbf340, seqs=0x2aab21e4a010, seqvals=0x2aab21e4d4b0, ids=<value optimized out>, idvals=<value optimized out>, numdocs=<value optimized out>)
at src/couch_save.c:228
#13 0x00002aaaab1ccbc0 in couchstore_save_documents (db=0xcfbf340, docs=0x1263de38, infos=<value optimized out>, numdocs=421, options=1) at src/couch_save.c:359
#14 0x00002aaaaaf6eb37 in CouchKVStore::saveDocs (this=0xd021600, vbid=436, rev=1, docs=0x3c530000, docinfos=0x1a990000, docCount=421) at src/couch-kvstore/couch-kvstore.cc:1556
#15 0x00002aaaaaf6f241 in CouchKVStore::commit2couchstore (this=0xd021600) at src/couch-kvstore/couch-kvstore.cc:1491
#16 0x00002aaaaaf6f559 in queueItem (this=0xd021600, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:1623
#17 CouchKVStore::set (this=0xd021600, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:344
#18 0x00002aaaaaefd114 in EventuallyPersistentStore::flushOneDelOrSet (this=0x1118ed80, qi=..., rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2412
#19 0x00002aaaaaefd3db in EventuallyPersistentStore::flushOne (this=0x1118ed80, queue=<value optimized out>, rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2460
#20 0x00002aaaaaf00ed5 in EventuallyPersistentStore::flushVBQueue (this=0x1118ed80, vb=..., vb_queue=std::queue wrapping: std::deque with 456 elements =
, vbid=437, data_age=0) at src/ep.cc:2014
#21 0x00002aaaaaf0212c in EventuallyPersistentStore::flushOutgoingQueue (this=0x1118ed80, flushQueue=0x1118f048, flushPhase=@0xcfaef90, nextVbid=@0xcfaef98) at src/ep.cc:1956
#22 0x00002aaaaaf2b55c in Flusher::doFlush (this=0xcfaeea0) at src/flusher.cc:245
#23 0x00002aaaaaf2c395 in Flusher::step (this=0xcfaeea0, d=..., tid=...) at src/flusher.cc:158
#24 0x00002aaaaaef459a in Dispatcher::run (this=0xd001c00) at src/dispatcher.cc:173
#25 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd001c00) at src/dispatcher.cc:28
#26 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#27 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x4cc47940 (LWP 17853)):
#0 VBucket::getBGFetchItems (this=0xdceec00, fetches=std::tr1::unordered_map with 0 elements) at src/vbucket.cc:212
#1 0x00002aaaaaee729b in BgFetcher::run (this=0x111ce2d0, tid=<value optimized out>) at src/bgfetcher.cc:109
#2 0x00002aaaaaef459a in Dispatcher::run (this=0xd001a40) at src/dispatcher.cc:173
#3 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd001a40) at src/dispatcher.cc:28
#4 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#5 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x4d648940 (LWP 17854)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
--Type <return> to continue, or q <return> to quit--#1 0x00002aaaaaef1ed8 in wait (this=0xd034ea0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0xd034ea0, d=...) at src/dispatcher.cc:336
#3 0x00002aaaaaef459a in Dispatcher::run (this=0xd001880) at src/dispatcher.cc:173
#4 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd001880) at src/dispatcher.cc:28
#5 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#6 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x4e049940 (LWP 17855)):
#0 0x00000033a420b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef1ed8 in wait (this=0xd035830, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0xd035830, d=...) at src/dispatcher.cc:336
#3 0x00002aaaaaef459a in Dispatcher::run (this=0xd0016c0) at src/dispatcher.cc:173
#4 0x00002aaaaaef4e9b in launch_dispatcher_thread (arg=0xd0016c0) at src/dispatcher.cc:28
#5 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#6 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x4ea4a940 (LWP 17856)):
#0 0x00000033a420d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00000033a4208e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x00000033a4208d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004155f6 in notify_io_complete (cookie=0xcec8dc0, status=ENGINE_SUCCESS) at daemon/thread.c:485
#4 0x00002aaaaaf4c9b9 in operator() (this=0xcfaed80) at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/bits/stl_function.h:457
#5 operator() (this=0xcfaed80) at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/backward/binders.h:153
#6 for_each<std::_List_iterator<void const*>, std::binder2nd<std::pointer_to_binary_function<void const*, ENGINE_ERROR_CODE, void> > > (this=0xcfaed80)
at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/bits/stl_algo.h:4200
#7 notifyIOComplete<std::list<void const*, std::allocator<void const*> > > (this=0xcfaed80) at src/ep_engine.h:443
#8 TapConnMap::notifyIOThreadMain (this=0xcfaed80) at src/tapconnmap.cc:494
#9 0x00002aaaaaf10cdc in EventuallyPersistentEngine::notifyPendingConnections (this=0xf046900) at src/ep_engine.cc:3408
#10 0x00002aaaaaf10e93 in EvpNotifyPendingConns (arg=0xf046900) at src/ep_engine.cc:1142
#11 0x00000033a420677d in start_thread () from /lib64/libpthread.so.0
#12 0x00000033a3ad3c1d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b633ae55220 (LWP 17029)):
#0 0x00000033a3ad4008 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b633a9da576 in epoll_dispatch (base=0xcfb8000, tv=<value optimized out>) at epoll.c:404
#2 0x00002b633a9c5e44 in event_base_loop (base=0xcfb8000, flags=<value optimized out>) at event.c:1558
#3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918
(gdb) quit
A debugging session is active.
Inferior 1 [process 17029] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
Detaching from program: /opt/couchbase/bin/memcached, process 17029