Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8068

memcached crashed when load items to bucket

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      centos 5.8 64bit

      Description

      Install couchbase server 2.0.2-760 on 2 nodes (49 and 50).
      Each node only has one default bucket
      Setup xdcr from 49 to 50
      Load item to node 49 using cbworkloadgen: ./cbworkloadgen -n localhost:8091 -r .9 -i 100000 -s 10 --threads=10 --prefix=test_

      In few seconds, memcached on node 49 crashed.

      Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 134. Restarting. Messages: Wed Apr 10 15:15:54.109673 PDT 3: (default) Persisting ``test_13511'' on vb512 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.109686 PDT 3: (default) Persisting ``test_14857'' on vb512 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.109697 PDT 3: (default) Persisting ``test_15153'' on vb512 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.109708 PDT 3: (default) Persisting ``test_15621'' on vb512 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.167117 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/533.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.167143 PDT 3: (default) Warning: failed to open database, vbucketId = 533 fileRev = 1 numDocs = 4
      Wed Apr 10 15:15:54.167151 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 533 rev = 1
      Wed Apr 10 15:15:54.167167 PDT 3: (default) Persisting ``test_13957'' on vb533 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.167180 PDT 3: (default) Persisting ``test_14363'' on vb533 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.167191 PDT 3: (default) Persisting ``test_14411'' on vb533 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.167202 PDT 3: (default) Persisting ``test_14581'' on vb533 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.201462 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/540.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.201493 PDT 3: (default) Warning: failed to open database, vbucketId = 540 fileRev = 1 numDocs = 5
      Wed Apr 10 15:15:54.201502 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 540 rev = 1
      Wed Apr 10 15:15:54.201518 PDT 3: (default) Persisting ``test_13671'' on vb540 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.201531 PDT 3: (default) Persisting ``test_15233'' on vb540 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.201542 PDT 3: (default) Persisting ``test_15541'' on vb540 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.201553 PDT 3: (default) Persisting ``test_16012'' on vb540 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.201564 PDT 3: (default) Persisting ``test_16182'' on vb540 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.250633 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/554.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.250653 PDT 3: (default) Warning: failed to open database, vbucketId = 554 fileRev = 1 numDocs = 5
      Wed Apr 10 15:15:54.250661 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 554 rev = 1
      Wed Apr 10 15:15:54.250677 PDT 3: (default) Persisting ``test_13520'' on vb554 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.250690 PDT 3: (default) Persisting ``test_14866'' on vb554 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.250701 PDT 3: (default) Persisting ``test_15162'' on vb554 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.250712 PDT 3: (default) Persisting ``test_15610'' on vb554 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.250723 PDT 3: (default) Persisting ``test_15780'' on vb554 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.293072 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/566.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.293092 PDT 3: (default) Warning: failed to open database, vbucketId = 566 fileRev = 1 numDocs = 5
      Wed Apr 10 15:15:54.293125 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 566 rev = 1
      Wed Apr 10 15:15:54.293144 PDT 3: (default) Persisting ``test_13640'' on vb566 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.293157 PDT 3: (default) Persisting ``test_15202'' on vb566 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.293170 PDT 3: (default) Persisting ``test_15392'' on vb566 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.293181 PDT 3: (default) Persisting ``test_15570'' on vb566 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.293192 PDT 3: (default) Persisting ``test_16023'' on vb566 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.334857 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/575.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.334879 PDT 3: (default) Warning: failed to open database, vbucketId = 575 fileRev = 1 numDocs = 3
      Wed Apr 10 15:15:54.334887 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 575 rev = 1
      Wed Apr 10 15:15:54.334910 PDT 3: (default) Persisting ``test_13966'' on vb575 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.334923 PDT 3: (default) Persisting ``test_14352'' on vb575 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.334935 PDT 3: (default) Persisting ``test_14420'' on vb575 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.387860 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/591.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.387881 PDT 3: (default) Warning: failed to open database, vbucketId = 591 fileRev = 1 numDocs = 4
      Wed Apr 10 15:15:54.387889 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 591 rev = 1
      Wed Apr 10 15:15:54.387906 PDT 3: (default) Persisting ``test_14072'' on vb591 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.387919 PDT 3: (default) Persisting ``test_14690'' on vb591 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.387930 PDT 3: (default) Persisting ``test_14700'' on vb591 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.387941 PDT 3: (default) Persisting ``test_15976'' on vb591 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.417107 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/595.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.417133 PDT 3: (default) Warning: failed to open database, vbucketId = 595 fileRev = 1 numDocs = 4
      Wed Apr 10 15:15:54.417143 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 595 rev = 1
      Wed Apr 10 15:15:54.417168 PDT 3: (default) Persisting ``test_13926'' on vb595 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.417190 PDT 3: (default) Persisting ``test_14282'' on vb595 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.417210 PDT 3: (default) Persisting ``test_14312'' on vb595 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.417222 PDT 3: (default) Persisting ``test_14460'' on vb595 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.521202 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/633.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.521224 PDT 3: (default) Warning: failed to open database, vbucketId = 633 fileRev = 1 numDocs = 4
      Wed Apr 10 15:15:54.521232 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 633 rev = 1
      Wed Apr 10 15:15:54.521250 PDT 3: (default) Persisting ``test_13887'' on vb633 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.521263 PDT 3: (default) Persisting ``test_13917'' on vb633 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.521274 PDT 3: (default) Persisting ``test_14323'' on vb633 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.521285 PDT 3: (default) Persisting ``test_14451'' on vb633 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.574301 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/650.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.574323 PDT 3: (default) Warning: failed to open database, vbucketId = 650 fileRev = 1 numDocs = 4
      Wed Apr 10 15:15:54.574334 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 650 rev = 1
      Wed Apr 10 15:15:54.574357 PDT 3: (default) Persisting ``test_13836'' on vb650 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.574370 PDT 3: (default) Persisting ``test_14202'' on vb650 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.574382 PDT 3: (default) Persisting ``test_14392'' on vb650 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.574393 PDT 3: (default) Persisting ``test_14570'' on vb650 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.640303 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/672.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.640328 PDT 3: (default) Warning: failed to open database, vbucketId = 672 fileRev = 1 numDocs = 4
      Wed Apr 10 15:15:54.640336 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 672 rev = 1
      Wed Apr 10 15:15:54.640359 PDT 3: (default) Persisting ``test_13807'' on vb672 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.640372 PDT 3: (default) Persisting ``test_13997'' on vb672 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.640383 PDT 3: (default) Persisting ``test_14233'' on vb672 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.640394 PDT 3: (default) Persisting ``test_14541'' on vb672 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.721120 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/700.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.721141 PDT 3: (default) Warning: failed to open database, vbucketId = 700 fileRev = 1 numDocs = 3
      Wed Apr 10 15:15:54.721149 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 700 rev = 1
      Wed Apr 10 15:15:54.721165 PDT 3: (default) Persisting ``test_14153'' on vb700 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.721178 PDT 3: (default) Persisting ``test_14621'' on vb700 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.721189 PDT 3: (default) Persisting ``test_15857'' on vb700 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.775355 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/716.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.775383 PDT 3: (default) Warning: failed to open database, vbucketId = 716 fileRev = 1 numDocs = 6
      Wed Apr 10 15:15:54.775396 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 716 rev = 1
      Wed Apr 10 15:15:54.775420 PDT 3: (default) Persisting ``test_13847'' on vb716 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.775467 PDT 3: (default) Persisting ``test_14273'' on vb716 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.775493 PDT 3: (default) Persisting ``test_14491'' on vb716 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.775511 PDT 3: (default) Persisting ``test_14501'' on vb716 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.775530 PDT 3: (default) Persisting ``test_16956'' on vb716 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.775548 PDT 3: (default) Persisting ``test_17052'' on vb716 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.804723 PDT 3: (default) Warning: couchstore_open_db failed, name=/opt/couchbase/var/lib/couchbase/data/default/720.couch.1 option=0 rev=1 error=no such file [none]
      Wed Apr 10 15:15:54.804745 PDT 3: (default) Warning: failed to open database, vbucketId = 720 fileRev = 1 numDocs = 5
      Wed Apr 10 15:15:54.804757 PDT 3: (default) Warning: commit failed, cannot save CouchDB docs for vbucket = 720 rev = 1
      Wed Apr 10 15:15:54.804774 PDT 3: (default) Persisting ``test_14083'' on vb720 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.804787 PDT 3: (default) Persisting ``test_14113'' on vb720 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.804798 PDT 3: (default) Persisting ``test_14661'' on vb720 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.804815 PDT 3: (default) Persisting ``test_15817'' on vb720 (rowid=-2) returned 0 updates
      Wed Apr 10 15:15:54.804833 PDT 3: (default) Persisting ``test_15987'' on vb720 (rowid=-2) returned 0 updates
      memcached: src/ep.cc:1781: virtual void PersistenceCallback::callback(mutation_result&): Assertion `stats->diskQueueSize < ((size_t)1<<(sizeof(size_t)*8-1))' failed. ns_log000 ns_1@127.0.0.1 15:16:19 - Wed Apr 10, 2013

      Check coredump, I see it terminate with signal 6 from /lib64/libc.so.6

      Core file callstacks:
      GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
      Copyright (C) 2009 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-redhat-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/bin/memcached...done.
      [New Thread 8090]
      [New Thread 8094]
      [New Thread 8093]
      [New Thread 8092]
      [New Thread 8091]
      [New Thread 8089]
      [New Thread 7929]
      [New Thread 7928]
      [New Thread 7927]
      [New Thread 7926]
      [New Thread 7925]
      [New Thread 7923]
      [New Thread 7922]
      [New Thread 7913]
      Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
      Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
      Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
      Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libdl.so.2
      Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libm.so.6
      Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/librt.so.1
      Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
      Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
      [Thread debugging using libthread_db enabled]
      Loaded symbols for /lib64/libpthread.so.0
      Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libstdc++.so.6
      Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libgcc_s.so.1
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
      Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
      Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
      Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libicudata.so.44
      Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

      warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff21bc2000
      Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
      Program terminated with signal 6, Aborted.
      #0 0x0000003866c30285 in raise () from /lib64/libc.so.6

      Thread 14 (Thread 0x2b1ca9935220 (LWP 7913)):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b1ca94bb576 in epoll_dispatch (base=0x12ae0000, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b1ca94a6e44 in event_base_loop (base=0x12ae0000, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7923

      Thread 13 (Thread 7922):
      #0 0x0000003866cc545b in read () from /lib64/libc.so.6
      #1 0x0000003866c6b677 in _IO_new_file_underflow () from /lib64/libc.so.6
      #2 0x0000003866c6c03e in _IO_default_uflow_internal () from /lib64/libc.so.6
      #3 0x0000003866c61124 in _IO_getline_info_internal () from /lib64/libc.so.6
      #4 0x0000003866c5ffc9 in fgets () from /lib64/libc.so.6
      #5 0x00002b1ca9936939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
      #6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #7 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 12 (Thread 7923):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0xe27c040) at extensions/loggers/file_logger.c:368
      #2 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #3 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 11 (Thread 7925):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b1ca94bb576 in epoll_dispatch (base=0x12ae0500, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b1ca94a6e44 in event_base_loop (base=0x12ae0500, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0xe27f900) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 10 (Thread 7926):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b1ca94bb576 in epoll_dispatch (base=0x12ae0280, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b1ca94a6e44 in event_base_loop (base=0x12ae0280, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0xe27f9f8) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 9 (Thread 7927):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b1ca94bb576 in epoll_dispatch (base=0x12ae0c80, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b1ca94a6e44 in event_base_loop (base=0x12ae0c80, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0xe27faf0) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 8 (Thread 7928):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b1ca94bb576 in epoll_dispatch (base=0x12ae0a00, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b1ca94a6e44 in event_base_loop (base=0x12ae0a00, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0xe27fbe8) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 7 (Thread 7929):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b1ca94bb576 in epoll_dispatch (base=0x12ae0780, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b1ca94a6e44 in event_base_loop (base=0x12ae0780, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0xe27fce0) at daemon/thread.c:301
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 6 (Thread 8089):
      #0 0x0000003866c99221 in nanosleep () from /lib64/libc.so.6
      #1 0x0000003866cccba4 in usleep () from /lib64/libc.so.6
      #2 0x00002aaaaaf2ee25 in updateStatsThread (arg=0xe27c4c0) at src/memory_tracker.cc:31
      #3 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #4 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 5 (Thread 8091):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29d8 in wait (this=0x12ad02d0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x12ad02d0, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef520a in Dispatcher::run (this=0x12b1ca80) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5abd in launch_dispatcher_thread (arg=0x12b1ca80) at src/dispatcher.cc:28
      #5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #6 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 4 (Thread 8092):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29d8 in wait (this=0x12ad0240, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x12ad0240, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef520a in Dispatcher::run (this=0x12b1d880) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5abd in launch_dispatcher_thread (arg=0x12b1d880) at src/dispatcher.cc:28
      #5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #6 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 3 (Thread 8093):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29d8 in wait (this=0x12ad01b0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x12ad01b0, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef520a in Dispatcher::run (this=0x12b1d6c0) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5abd in launch_dispatcher_thread (arg=0x12b1d6c0) at src/dispatcher.cc:28
      #5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #6 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 2 (Thread 8094):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf0ff0f in wait (this=0x12adc000) at src/syncobject.hh:57
      #2 wait (this=0x12adc000) at src/syncobject.hh:73
      #3 wait (this=0x12adc000) at src/tapconnmap.hh:169
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x12adc000) at src/ep_engine.cc:3359
      #5 0x00002aaaaaf0fff3 in EvpNotifyPendingConns (arg=0x12adc000) at src/ep_engine.cc:1121
      #6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #7 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x47120940 (LWP 8090)):
      #0 0x0000003866c30285 in raise () from /lib64/libc.so.6
      #1 0x0000003866c31d30 in abort () from /lib64/libc.so.6
      #2 0x0000003866c29706 in __assert_fail () from /lib64/libc.so.6
      #3 0x00002aaaaaf0cf4a in PersistenceCallback::callback(std::pair<int, long>&) () from /opt/couchbase/lib/memcached/ep.so
      #4 0x00002aaaaaf62ee4 in CouchKVStore::commitCallback (this=0x12b48000, committedReqs=<value optimized out>, numReqs=3, errCode=COUCHSTORE_SUCCESS)
      at src/couch-kvstore/couch-kvstore.cc:1596
      #5 0x00002aaaaaf68acc in CouchKVStore::commit2couchstore (this=0x12b48000) at src/couch-kvstore/couch-kvstore.cc:1429
      #6 0x00002aaaaaf68caa in CouchKVStore::commit (this=0x12b48000) at src/couch-kvstore/couch-kvstore.cc:812
      #7 0x00002aaaaaf00c36 in EventuallyPersistentStore::flushVBucket (this=0x12ad8480, vbid=<value optimized out>) at src/ep.cc:1966
      #8 0x00002aaaaaf2a31a in doFlush (this=0x12ad2480, d=..., tid=...) at src/flusher.cc:214
      #9 Flusher::step (this=0x12ad2480, d=..., tid=...) at src/flusher.cc:153
      #10 0x00002aaaaaef520a in Dispatcher::run (this=0x12b1cc40) at src/dispatcher.cc:171
      #11 0x00002aaaaaef5abd in launch_dispatcher_thread (arg=0x12b1cc40) at src/dispatcher.cc:28
      #12 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #13 0x0000003866cd325d in clone () from /lib64/libc.so.6
      --------------------------------------------------------------------------------
      Module information:
      /opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
      lrwxrwxrwx 1 bin bin 31 Apr 10 13:40 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
      85a524f3c13a608b44f46a11f3458d3c /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
      /opt/couchbase/lib/libevent-2.0.so.5:
      lrwxrwxrwx 1 bin bin 21 Apr 10 13:40 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
      88902c21296f869ab9032f961ed3b4f8 /opt/couchbase/lib/libevent-2.0.so.5

      I will upload cbcollect_info soon

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Show
        thuan Thuan Nguyen added a comment - Link to memcached coredump https://s3.amazonaws.com/bugdb/jira/MB-8068/core.memcached.7913 Link to collect info of node 49 https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_04/cbcollect_node49.zip Link to collect info of node 50 https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_04/cbcollect_node50.zip Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-760-rel.rpm.manifest.xml
        Hide
        mikew Mike Wiederhold added a comment -

        Duplicate of MB-7938.

        Show
        mikew Mike Wiederhold added a comment - Duplicate of MB-7938 .
        Hide
        maria Maria McDuff (Inactive) added a comment -
        Show
        maria Maria McDuff (Inactive) added a comment - MB-7938

          People

          • Assignee:
            mikew Mike Wiederhold
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes