Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8243

memcached crashed in Configuration::getCouchBucket

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.1.0
    • 2.1.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • 2.0.2-795
    • Ubuntu 64-bit

    Description

      I see crash from http://qa.hq.northscale.net/job/CouchbaseServer-SanityTest-4Node-
      Centos64/63/consoleFull run on build 2.0.2-795-rel

      seems it's the test:
      ./testrunner -i fournode.ini -t viewquerytests.ViewQueryTests.test_employee_dataset_all_queries,limit=1000,docs-per-day=2,wait_persistence=true

      but this job doesn't generate collect info after failures, and I have logs that have been obtained much later

      gdb /opt/couchbase/bin/memcached core.memcached.7524
      GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
      Copyright (C) 2009 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-redhat-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/bin/memcached...done.
      [New Thread 7705]
      [New Thread 8584]
      [New Thread 8583]
      [New Thread 8582]
      [New Thread 7709]
      [New Thread 7708]
      [New Thread 7707]
      [New Thread 7706]
      [New Thread 7704]
      [New Thread 7703]
      [New Thread 7702]
      [New Thread 7701]
      [New Thread 7539]
      [New Thread 7538]
      [New Thread 7537]
      [New Thread 7536]
      [New Thread 7535]
      [New Thread 7533]
      [New Thread 7532]
      [New Thread 7524]

      warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the expected address

      warning: difference appears to be caused by prelink, adjusting expectations

      warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address

      warning: difference appears to be caused by prelink, adjusting expectations
      Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
      Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
      Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
      Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libdl.so.2
      Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libm.so.6
      Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/librt.so.1
      Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
      Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
      [Thread debugging using libthread_db enabled]
      Loaded symbols for /lib64/libpthread.so.0
      Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libstdc++.so.6
      Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libgcc_s.so.1
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
      Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
      Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
      Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libicudata.so.44
      Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

      warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff9dbe8000
      Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
      Program terminated with signal 6, Aborted.
      #0 0x0000003866c30285 in raise () from /lib64/libc.so.6
      (gdb) t aa bt
      A syntax error in expression, near `bt'.
      (gdb) t a a bt

      Thread 20 (Thread 0x2af656d41220 (LWP 7524)):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6000, tv=<value optimized out>) at epoll.c:404
      #2 0x00002af6568b2e44 in event_base_loop (base=0x222b6000, flags=<value optimized out>) at event.c:1558
      #3 0x00000000004097d6 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

      Thread 19 (Thread 7532):
      #0 0x0000003866cc545b in read () from /lib64/libc.so.6
      #1 0x0000003866c6b677 in _IO_new_file_underflow () from /lib64/libc.so.6
      #2 0x0000003866c6c03e in _IO_default_uflow_internal () from /lib64/libc.so.6
      #3 0x0000003866c61124 in _IO_getline_info_internal () from /lib64/libc.so.6
      #4 0x0000003866c5ffc9 in fgets () from /lib64/libc.so.6
      #5 0x00002af656d42939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
      #6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #7 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 18 (Thread 7533):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x1da4e040) at extensions/loggers/file_logger.c:368
      #2 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #3 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 17 (Thread 7535):
      #0 0x00002af656b17f6f in tcmalloc::CentralFreeList::Populate() () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #1 0x00002af656b17c93 in tcmalloc::CentralFreeList::FetchFromSpansSafe() () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #2 0x00002af656b17bcc in tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #3 0x00002af656b1ccd5 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #4 0x00002af656b12054 in tcmalloc::ThreadCache::Allocate(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #5 0x00002af656b10b60 in (anonymous namespace)::do_malloc(unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #6 0x00002af656b10892 in (anonymous namespace)::do_malloc_or_cpp_alloc(unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #7 0x00002af656b10bec in (anonymous namespace)::do_calloc(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #8 0x00002af656b24764 in tc_calloc () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
      #9 0x00002aaaaaef94b9 in HashTable (this=<value optimized out>, i=918, newState=vbucket_state_replica, st=..., checkpointConfig=..., checkpointId=1, initState=vbucket_state_dead)
      at src/stored-value.hh:814
      #10 VBucket::VBucket (this=<value optimized out>, i=918, newState=vbucket_state_replica, st=..., checkpointConfig=..., checkpointId=1, initState=vbucket_state_dead) at src/vbucket.hh:126
      #11 0x00002aaaaaefb932 in EventuallyPersistentStore::setVBucketState (this=0x2a424400, vbid=918, to=vbucket_state_replica) at src/ep.cc:795
      #12 0x00002aaaaaf1e11d in setVBucketState (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.h:508
      #13 setVBucket (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:738
      #14 processUnknownCommand (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:884
      #15 0x00002aaaaaf1e99c in EvpUnknownCommand (handle=<value optimized out>, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:1021
      #16 0x00002aaaaacc4dd4 in bucket_unknown_command (handle=<value optimized out>, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at bucket_engine.c:2475
      #17 0x0000000000411b4e in process_bin_unknown_packet (c=0x222858c0) at daemon/memcached.c:2882
      #18 process_bin_packet (c=0x222858c0) at daemon/memcached.c:3170
      #19 complete_nread_binary (c=0x222858c0) at daemon/memcached.c:3744
      #20 complete_nread (c=0x222858c0) at daemon/memcached.c:3826
      #21 conn_nread (c=0x222858c0) at daemon/memcached.c:5679
      #22 0x0000000000405ec5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x222858c0) at daemon/memcached.c:5942
      #23 0x00002af6568b2f3c in event_process_active_single_queue (base=0x222b6500, flags=<value optimized out>) at event.c:1308
      #24 event_process_active (base=0x222b6500, flags=<value optimized out>) at event.c:1375
      #25 event_base_loop (base=0x222b6500, flags=<value optimized out>) at event.c:1572
      #26 0x0000000000414604 in worker_libevent (arg=0x1da51900) at daemon/thread.c:301
      #27 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #28 0x0000003866cd325d in clone () from /lib64/libc.so.6

      --Type <return> to continue, or q <return> to quit--
      Thread 16 (Thread 7536):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6280, tv=<value optimized out>) at epoll.c:404
      #2 0x00002af6568b2e44 in event_base_loop (base=0x222b6280, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414604 in worker_libevent (arg=0x1da519f8) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 15 (Thread 7537):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6c80, tv=<value optimized out>) at epoll.c:404
      #2 0x00002af6568b2e44 in event_base_loop (base=0x222b6c80, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414604 in worker_libevent (arg=0x1da51af0) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 14 (Thread 7538):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6a00, tv=<value optimized out>) at epoll.c:404
      #2 0x00002af6568b2e44 in event_base_loop (base=0x222b6a00, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414604 in worker_libevent (arg=0x1da51be8) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 13 (Thread 7539):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6780, tv=<value optimized out>) at epoll.c:404
      #2 0x00002af6568b2e44 in event_base_loop (base=0x222b6780, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414604 in worker_libevent (arg=0x1da51ce0) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 12 (Thread 7701):
      #0 0x0000003866c99221 in nanosleep () from /lib64/libc.so.6
      #1 0x0000003866cccba4 in usleep () from /lib64/libc.so.6
      #2 0x00002aaaaaf35125 in updateStatsThread (arg=0x1da4e4c0) at src/memory_tracker.cc:31
      #3 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #4 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 11 (Thread 7702):
      #0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
      #2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
      #4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
      #5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
      #6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
      #7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26595800, vbucketId=936, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
      #8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26595800, vbstates=Traceback (most recent call last):
      File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
      nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
      RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
      std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
      #9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
      #10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
      --Type <return> to continue, or q <return> to quit--
      #11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361ba0) at src/scheduler.cc:153
      #12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361ba0) at src/scheduler.cc:34
      #13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #14 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 10 (Thread 7703):
      #0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
      #2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
      #4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
      #5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
      #6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
      #7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26595200, vbucketId=937, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
      #8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26595200, vbstates=Traceback (most recent call last):
      File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
      nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
      RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
      std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
      #9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
      #10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
      #11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361a00) at src/scheduler.cc:153
      #12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361a00) at src/scheduler.cc:34
      #13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #14 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 9 (Thread 7704):
      #0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
      #2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
      #4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
      #5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
      #6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
      #7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26594c00, vbucketId=934, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
      #8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26594c00, vbstates=Traceback (most recent call last):
      File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
      nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
      RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
      std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
      #9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
      #10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
      #11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361860) at src/scheduler.cc:153
      #12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361860) at src/scheduler.cc:34
      #13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #14 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 8 (Thread 7706):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf39631 in wait (this=0x22388d00) at src/syncobject.hh:57
      #2 ExecutorThread::run (this=0x22388d00) at src/scheduler.cc:139
      #3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388d00) at src/scheduler.cc:34
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      --Type <return> to continue, or q <return> to quit--
      Thread 7 (Thread 7707):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf39631 in wait (this=0x22388b60) at src/syncobject.hh:57
      #2 ExecutorThread::run (this=0x22388b60) at src/scheduler.cc:139
      #3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388b60) at src/scheduler.cc:34
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 6 (Thread 7708):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf39631 in wait (this=0x223889c0) at src/syncobject.hh:57
      #2 ExecutorThread::run (this=0x223889c0) at src/scheduler.cc:139
      #3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x223889c0) at src/scheduler.cc:34
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 5 (Thread 7709):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf39631 in wait (this=0x22388820) at src/syncobject.hh:57
      #2 ExecutorThread::run (this=0x22388820) at src/scheduler.cc:139
      #3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388820) at src/scheduler.cc:34
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 4 (Thread 8582):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf0fe0f in wait (this=0x2230e900) at src/syncobject.hh:57
      #2 wait (this=0x2230e900) at src/syncobject.hh:73
      #3 wait (this=0x2230e900) at src/tapconnmap.hh:169
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x2230e900) at src/ep_engine.cc:3379
      #5 0x00002aaaaaf0fef3 in EvpNotifyPendingConns (arg=0x2230e900) at src/ep_engine.cc:1153
      #6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #7 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 3 (Thread 8583):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef3688 in wait (this=0x222aa1b0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x222aa1b0, d=...) at src/dispatcher.cc:342
      #3 0x00002aaaaaef61ea in Dispatcher::run (this=0x22313c00) at src/dispatcher.cc:184
      #4 0x00002aaaaaef69ad in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
      #5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #6 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 2 (Thread 8584):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef3688 in wait (this=0x222ab0e0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x222ab0e0, d=...) at src/dispatcher.cc:342
      #3 0x00002aaaaaef61ea in Dispatcher::run (this=0x22313500) at src/dispatcher.cc:184
      #4 0x00002aaaaaef69ad in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
      #5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #6 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x49688940 (LWP 7705)):
      #0 0x0000003866c30285 in raise () from /lib64/libc.so.6
      #1 0x0000003866c31d30 in abort () from /lib64/libc.so.6
      --Type <return> to continue, or q <return> to quit--
      #2 0x00002aaaaaf36070 in Mutex::acquire (this=0x2230e828) at src/mutex.cc:83
      #3 0x00002aaaaaf83fe8 in lock (this=0x2230e828, key="couch_bucket") at src/locks.hh:48
      #4 LockHolder (this=0x2230e828, key="couch_bucket") at src/locks.hh:26
      #5 Configuration::getString (this=0x2230e828, key="couch_bucket") at src/configuration.cc:38
      #6 0x00002aaaaaf8e5eb in Configuration::getCouchBucket (this=0x2230e828) at src/generated_configuration.cc:71
      #7 0x00002aaaaaf7c59e in CouchNotifier::selectBucket (this=0x22331000) at src/couch-kvstore/couch-notifier.cc:721
      #8 0x00002aaaaaf7cc0f in CouchNotifier::processInput (this=0x22331000) at src/couch-kvstore/couch-notifier.cc:606
      #9 0x00002aaaaaf7c199 in maybeProcessInput (this=0x22331000, rh=0x22385540) at src/couch-kvstore/couch-notifier.cc:546
      #10 CouchNotifier::sendCommand (this=0x22331000, rh=0x22385540) at src/couch-kvstore/couch-notifier.cc:439
      #11 0x00002aaaaaf7d478 in CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:774
      #12 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26594600, vbucketId=935, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
      #13 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26594600, vbstates=Traceback (most recent call last):
      File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
      nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
      RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
      std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
      #14 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
      #15 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
      #16 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22388ea0) at src/scheduler.cc:153
      #17 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388ea0) at src/scheduler.cc:34
      #18 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #19 0x0000003866cd325d in clone () from /lib64/libc.so.6

      core file:
      root(couchbase)@10.3.3.30:/tmp/core.memcached.7524

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jin Jin Lim (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty