Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7509

memcached crashed in EventuallyPersistentStore

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.0.1
    • 2.0.1
    • couchbase-bucket
    • Security Level: Public
    • None

    Description

      it a simple test rebalance test with doc ops:
      ./testrunner -i /tmp/rebalance-tests.ini get-logs=True,wait_timeout=180,GROUP=P0 -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,GROUP=IN;P0
      http://qa.hq.northscale.net/view/2.0.1/job/ubuntu-64-2.0-new-rebalance-tests-P0/310/consoleFull

      on node 10.1.3.121

      gdb /opt/couchbase/bin/memcached core.memcached.22487
      GNU gdb (GDB) 7.1-ubuntu
      Copyright (C) 2010 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/bin/memcached...done.

      warning: exec file is newer than core file.
      [New Thread 22504]
      [New Thread 22505]
      [New Thread 22502]
      [New Thread 22506]
      [New Thread 22507]
      [New Thread 22498]
      [New Thread 22500]
      [New Thread 22503]
      [New Thread 22501]
      [New Thread 22499]
      [New Thread 22497]
      [New Thread 22496]
      [New Thread 22487]
      [New Thread 22508]

      warning: Can't read pathname for load map: Input/output error.

      warning: .dynamic section for "/opt/couchbase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?)

      warning: .dynamic section for "/opt/couchbase/lib/libcouchstore.so.1" is not at the expected address (wrong library or version mismatch?)
      Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
      Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
      Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
      Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib/libdl.so.2
      Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/libm.so.6
      Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/librt.so.1
      Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
      Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
      Loaded symbols for /lib/libpthread.so.0
      Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib/libstdc++.so.6
      Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libgcc_s.so.1
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libz.so.1
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
      Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
      Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib/libnss_files.so.2
      Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
      Program terminated with signal 11, Segmentation fault.
      #0 0x00007f67c84a024b in read_id_record (in=0x68eac90, buf=0x0, ctx=<value optimized out>) at src/db_compact.c:93
      93 src/db_compact.c: No such file or directory.
      in src/db_compact.c
      (gdb) t a a bt

      Thread 14 (Thread 22508):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67c86f800f in processUnknownCommand (h=<value optimized out>, cookie=<value optimized out>, request=<value optimized out>, response=<value optimized out>) at src/ep_engine.cc:1005
      #2 0x0000000000000000 in ?? ()

      Thread 13 (Thread 22487):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4000, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4000, flags=<value optimized out>) at event.c:1558
      #3 0x000000000040c2e1 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918

      Thread 12 (Thread 22496):
      #0 0x00007f67cd0ad4bd in read () from /lib/libc.so.6
      #1 0x00007f67cd048348 in _IO_file_underflow () from /lib/libc.so.6
      #2 0x00007f67cd049eee in _IO_default_uflow () from /lib/libc.so.6
      #3 0x00007f67cd03e43e in _IO_getline_info () from /lib/libc.so.6
      #4 0x00007f67cd03d329 in fgets () from /lib/libc.so.6
      #5 0x00007f67cc8a7b19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
      #6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
      #7 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #8 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #9 0x0000000000000000 in ?? ()

      Thread 11 (Thread 22497):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67cbea3e16 in add_log_entry (severity=<value optimized out>, client_cookie=<value optimized out>, fmt=0x1672000 "\023\020u\001") at extensions/loggers/file_logger.c:188
      #2 logger_log (severity=<value optimized out>, client_cookie=<value optimized out>, fmt=0x1672000 "\023\020u\001") at extensions/loggers/file_logger.c:280
      #3 0x0000000000000000 in ?? ()

      Thread 10 (Thread 22499):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4280, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4280, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x12) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 9 (Thread 22501):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4a00, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4a00, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x1e) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 8 (Thread 22503):
      #0 0x00007f67cd07f39d in nanosleep () from /lib/libc.so.6
      #1 0x00007f67cd0b4844 in usleep () from /lib/libc.so.6
      #2 0x00007f67c87183d5 in TapProducer::getQueueSize_UNLOCKED (this=0x7f67c828fe70) at src/tapconnection.cc:2059
      #3 TapProducer::getBackfillQueueSize_UNLOCKED (this=0x7f67c828fe70) at src/tapconnection.cc:2054
      #4 0x0000000000000000 in ?? ()

      --Type <return> to continue, or q <return> to quit--
      Thread 7 (Thread 22500):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4c80, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4c80, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x18) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 6 (Thread 22498):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4500, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4500, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0xc) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 5 (Thread 22507):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

      #1 0x00007f67c86d41a0 in EventuallyPersistentStore::bgFetch (this=<value optimized out>, key=..., vbucket=<value optimized out>, rowid=1310, cookie=0x7f67c607ee30, type=<value optimized out>)
      at src/ep.cc:1250
      #2 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #3 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #4 0x0000000000000000 in ?? ()

      Thread 4 (Thread 22506):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67c86d41a0 in EventuallyPersistentStore::bgFetch (this=<value optimized out>, key=..., vbucket=<value optimized out>, rowid=4366, cookie=0x7f67c687fe30, type=<value optimized out>)
      at src/ep.cc:1250
      #2 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #3 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #4 0x0000000000000000 in ?? ()

      Thread 3 (Thread 22502):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4780, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4780, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x24) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 2 (Thread 22505):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67c86d41a0 in EventuallyPersistentStore::bgFetch (this=<value optimized out>, key=..., vbucket=<value optimized out>, rowid=79, cookie=0x7f67c7080e30, type=<value optimized out>)
      at src/ep.cc:1250
      #2 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #3 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #4 0x0000000000000000 in ?? ()

      Thread 1 (Thread 22504):
      #0 0x00007f67c84a024b in read_id_record (in=0x68eac90, buf=0x0, ctx=<value optimized out>) at src/db_compact.c:93
      #1 0x0000000000000000 in ?? ()

      Attachments

        For Gerrit Dashboard: MB-7509
        # Subject Branch Project Status CR V

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty