Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7509

memcached crashed in EventuallyPersistentStore

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None

      Description

      it a simple test rebalance test with doc ops:
      ./testrunner -i /tmp/rebalance-tests.ini get-logs=True,wait_timeout=180,GROUP=P0 -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,GROUP=IN;P0
      http://qa.hq.northscale.net/view/2.0.1/job/ubuntu-64-2.0-new-rebalance-tests-P0/310/consoleFull

      on node 10.1.3.121

      gdb /opt/couchbase/bin/memcached core.memcached.22487
      GNU gdb (GDB) 7.1-ubuntu
      Copyright (C) 2010 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/bin/memcached...done.

      warning: exec file is newer than core file.
      [New Thread 22504]
      [New Thread 22505]
      [New Thread 22502]
      [New Thread 22506]
      [New Thread 22507]
      [New Thread 22498]
      [New Thread 22500]
      [New Thread 22503]
      [New Thread 22501]
      [New Thread 22499]
      [New Thread 22497]
      [New Thread 22496]
      [New Thread 22487]
      [New Thread 22508]

      warning: Can't read pathname for load map: Input/output error.

      warning: .dynamic section for "/opt/couchbase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?)

      warning: .dynamic section for "/opt/couchbase/lib/libcouchstore.so.1" is not at the expected address (wrong library or version mismatch?)
      Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
      Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
      Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
      Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib/libdl.so.2
      Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/libm.so.6
      Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/librt.so.1
      Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
      Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
      Loaded symbols for /lib/libpthread.so.0
      Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib/libstdc++.so.6
      Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libgcc_s.so.1
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libz.so.1
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
      Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
      Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib/libnss_files.so.2
      Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
      Program terminated with signal 11, Segmentation fault.
      #0 0x00007f67c84a024b in read_id_record (in=0x68eac90, buf=0x0, ctx=<value optimized out>) at src/db_compact.c:93
      93 src/db_compact.c: No such file or directory.
      in src/db_compact.c
      (gdb) t a a bt

      Thread 14 (Thread 22508):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67c86f800f in processUnknownCommand (h=<value optimized out>, cookie=<value optimized out>, request=<value optimized out>, response=<value optimized out>) at src/ep_engine.cc:1005
      #2 0x0000000000000000 in ?? ()

      Thread 13 (Thread 22487):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4000, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4000, flags=<value optimized out>) at event.c:1558
      #3 0x000000000040c2e1 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918

      Thread 12 (Thread 22496):
      #0 0x00007f67cd0ad4bd in read () from /lib/libc.so.6
      #1 0x00007f67cd048348 in _IO_file_underflow () from /lib/libc.so.6
      #2 0x00007f67cd049eee in _IO_default_uflow () from /lib/libc.so.6
      #3 0x00007f67cd03e43e in _IO_getline_info () from /lib/libc.so.6
      #4 0x00007f67cd03d329 in fgets () from /lib/libc.so.6
      #5 0x00007f67cc8a7b19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
      #6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
      #7 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #8 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #9 0x0000000000000000 in ?? ()

      Thread 11 (Thread 22497):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67cbea3e16 in add_log_entry (severity=<value optimized out>, client_cookie=<value optimized out>, fmt=0x1672000 "\023\020u\001") at extensions/loggers/file_logger.c:188
      #2 logger_log (severity=<value optimized out>, client_cookie=<value optimized out>, fmt=0x1672000 "\023\020u\001") at extensions/loggers/file_logger.c:280
      #3 0x0000000000000000 in ?? ()

      Thread 10 (Thread 22499):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4280, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4280, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x12) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 9 (Thread 22501):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4a00, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4a00, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x1e) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 8 (Thread 22503):
      #0 0x00007f67cd07f39d in nanosleep () from /lib/libc.so.6
      #1 0x00007f67cd0b4844 in usleep () from /lib/libc.so.6
      #2 0x00007f67c87183d5 in TapProducer::getQueueSize_UNLOCKED (this=0x7f67c828fe70) at src/tapconnection.cc:2059
      #3 TapProducer::getBackfillQueueSize_UNLOCKED (this=0x7f67c828fe70) at src/tapconnection.cc:2054
      #4 0x0000000000000000 in ?? ()

      --Type <return> to continue, or q <return> to quit--
      Thread 7 (Thread 22500):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4c80, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4c80, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x18) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 6 (Thread 22498):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4500, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4500, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0xc) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 5 (Thread 22507):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

      #1 0x00007f67c86d41a0 in EventuallyPersistentStore::bgFetch (this=<value optimized out>, key=..., vbucket=<value optimized out>, rowid=1310, cookie=0x7f67c607ee30, type=<value optimized out>)
      at src/ep.cc:1250
      #2 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #3 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #4 0x0000000000000000 in ?? ()

      Thread 4 (Thread 22506):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67c86d41a0 in EventuallyPersistentStore::bgFetch (this=<value optimized out>, key=..., vbucket=<value optimized out>, rowid=4366, cookie=0x7f67c687fe30, type=<value optimized out>)
      at src/ep.cc:1250
      #2 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #3 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #4 0x0000000000000000 in ?? ()

      Thread 3 (Thread 22502):
      #0 0x00007f67cd0bbd03 in epoll_wait () from /lib/libc.so.6
      #1 0x00007f67cde74f36 in epoll_dispatch (base=0x5ef4780, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f67cde60394 in event_base_loop (base=0x5ef4780, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414ba4 in create_notification_pipe (me=0x24) at daemon/thread.c:190
      #4 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 2 (Thread 22505):
      #0 0x00007f67cd363bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00007f67c86d41a0 in EventuallyPersistentStore::bgFetch (this=<value optimized out>, key=..., vbucket=<value optimized out>, rowid=79, cookie=0x7f67c7080e30, type=<value optimized out>)
      at src/ep.cc:1250
      #2 0x00007f67cd35e9ca in start_thread () from /lib/libpthread.so.0
      #3 0x00007f67cd0bb70d in clone () from /lib/libc.so.6
      #4 0x0000000000000000 in ?? ()

      Thread 1 (Thread 22504):
      #0 0x00007f67c84a024b in read_id_record (in=0x68eac90, buf=0x0, ctx=<value optimized out>) at src/db_compact.c:93
      #1 0x0000000000000000 in ?? ()

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        jin Jin Lim added a comment -

        Andrei and Farshid, just for the bug maintenance would you please verify that this issue has been seen on 32-bit systems only?

        Show
        jin Jin Lim added a comment - Andrei and Farshid, just for the bug maintenance would you please verify that this issue has been seen on 32-bit systems only?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        yes. I confirmed with andrei that this is only seen on 32-bit system.
        he is now running same suite on 64-bit and should have results tomorrow

        Show
        farshid Farshid Ghods (Inactive) added a comment - yes. I confirmed with andrei that this is only seen on 32-bit system. he is now running same suite on 64-bit and should have results tomorrow
        Hide
        mikew Mike Wiederhold added a comment -

        Andrei,

        I just merged a fix that might solve this problem. This bug also has too many comments on it now so if you see another crash please file a new issue and assign it to me.

        Show
        mikew Mike Wiederhold added a comment - Andrei, I just merged a fix that might solve this problem. This bug also has too many comments on it now so if you see another crash please file a new issue and assign it to me.
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        all swap tests passed on ubuntu 32, centos 32, centos 64 against 161 build

        Show
        andreibaranouski Andrei Baranouski added a comment - all swap tests passed on ubuntu 32, centos 32, centos 64 against 161 build
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #481 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/481/)
        MB-7509: Skip merging checkpoint end items in chk queues (Revision b93e048a4321e9891481693ba8139826e9c97314)
        MB-7509 Removed item should be removed from keyIndex too. (Revision b1308a500e9610bb15379de57247b5e53104e1ba)

        Result = SUCCESS
        Mike Wiederhold :
        Files :

        • src/checkpoint.cc

        xiaoqin :
        Files :

        • src/checkpoint.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #481 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/481/ ) MB-7509 : Skip merging checkpoint end items in chk queues (Revision b93e048a4321e9891481693ba8139826e9c97314) MB-7509 Removed item should be removed from keyIndex too. (Revision b1308a500e9610bb15379de57247b5e53104e1ba) Result = SUCCESS Mike Wiederhold : Files : src/checkpoint.cc xiaoqin : Files : src/checkpoint.cc

          People

          • Assignee:
            mikew Mike Wiederhold
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes