Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-3764

memcached core dump in bucket_engine.c:1977

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.7 alpha 2
    • Fix Version/s: 1.7 beta
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      basestar-119
      Version: basestar-223-g75d647b

      Description

      spoke a little too soon about the build

      seeing this crash dump during regrssion on basestar-119

      Thread 1 (Thread 4736):
      #0 0x00007fbff149ea75 in raise () from /lib/libc.so.6
      #1 0x00007fbff14a25c0 in abort () from /lib/libc.so.6
      #2 0x00007fbff1497941 in __assert_fail () from /lib/libc.so.6
      #3 0x00007fbff0062cd4 in bucket_engine_release_cookie (cookie=0x407a8e8)
      at bucket_engine.c:1977
      #4 0x00007fbfed5583f3 in TapConnMap::notifyIOThreadMain (this=0x19a1c00)
      at tapconnmap.cc:441
      #5 0x00007fbfed502ff2 in EventuallyPersistentEngine::notifyTapIoThread (
      this=0x19a1660) at ep_engine.cc:3568
      #6 0x00007fbfed508a67 in EvpNotifyTapIo (arg=0x19a1660) at ep_engine.cc:882
      #7 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0
      #8 0x00007fbff15516cd in clone () from /lib/libc.so.6
      #9 0x0000000000000000 in ?? ()
      --------------------------------------

      core logs available also attached

      the core dump is pretty huge i dont think i can upload it to s3 or sth.

      i will take a look tomorrow to see which test triggered this crash...

      1. core-10.1.5.62-0.log
        28 kB
        Farshid Ghods
      2. mycore.memcached.20386.log
        12 kB
        Farshid Ghods
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        trond Trond Norbye added a comment -

        (gdb) print (struct conn)(cookie)
        $1 = {sfd = -1, nevents = 50, sasl_conn = 0x8ef9350, state = 0x406880 <conn_immediate_close>,
        substate = bin_reading_packet, registered_in_libevent = false, event = {ev_active_next =

        { tqe_next = 0x0, tqe_prev = 0x19b88d0}

        , ev_next =

        {tqe_next = 0x0, tqe_prev = 0x4b8ef48}

        ,
        ev_timeout_pos = {ev_next_with_common_timeout =

        {tqe_next = 0xffffffff, tqe_prev = 0x0}

        ,
        min_heap_idx = -1}, ev_fd = 83, ev_base = 0x19b84a0, _ev = {ev_io = {ev_io_next =

        {tqe_next = 0x0, tqe_prev = 0x7fbfe443bcd0}

        , ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {
        ev_signal_next =

        {tqe_next = 0x0, tqe_prev = 0x7fbfe443bcd0}

        , ev_ncalls = 0, ev_pncalls = 0x0}},
        ev_events = 22, ev_res = 0, ev_flags = 128, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout =

        { tv_sec = 0, tv_usec = 0}

        , ev_callback = 0x406fa0 <event_handler>, ev_arg = 0x407a8e8},
        ev_flags = 22, which = 2, rbuf = 0xe4342e0 "\201A", rcurr = 0xe4342e0 "\201A", rsize = 2048,
        rbytes = 0, wbuf = 0x407b3a0 "\200A", wcurr = 0x407b3a0 "\200A", wsize = 2048, wbytes = 78272,
        write_and_go = 0x412760 <conn_ship_log>, write_and_free = 0x0, ritem = 0xe4342f8 "\201D", rlbytes = 0,
        item = 0x0, store_op = 0, sbytes = 0, iov = 0x407c2b0, iovsize = 400, iovused = 0, msglist = 0x407dbc0,
        msgsize = 10, msgused = 1, msgcurr = 0, msgbytes = 0, ilist = 0x407bbb0, isize = 200,
        icurr = 0x407bbb0, ileft = 0, suffixlist = 0x407c200, suffixsize = 20, suffixcurr = 0x407c200,
        suffixleft = 0, protocol = binary_prot, transport = tcp_transport, request_id = 0, request_addr =

        { ss_family = 0, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}

        , request_addr_size = 0,
        hdrbuf = 0x0, hdrsize = 0, noreply = false, refcount = 1 '\001', dynamic_buffer =

        {buffer = 0x0, size = 2048, offset = 32}

        , engine_storage = 0x8ef9370, ascii_cmd = 0x0, binary_header = {request =

        { magic = 129 '\201', opcode = 65 'A', keylen = 0, extlen = 0 '\000', datatype = 0 '\000', vbucket = 0, bodylen = 0, opaque = 644349952, cas = 0}

        ,
        bytes = "\201A", '\000' <repeats 12 times>, "h&\000\000\000\000\000\000\000"}, cas = 0, cmd = 65,
        opaque = 644349952, keylen = 0, list_state = 0, next = 0x0, thread = 0x19b5fc0,
        aiostat = ENGINE_SUCCESS, ewouldblock = true, tap_nack_mode = true,
        tap_iterator = 0x7fbff0061789 <bucket_tap_iterator_shim>}

        The refcount is 1 here meaning that the tap thread shouldn't try to reference this object at all (that's the core's reference)...

        Show
        trond Trond Norbye added a comment - (gdb) print (struct conn )(cookie) $1 = {sfd = -1, nevents = 50, sasl_conn = 0x8ef9350, state = 0x406880 <conn_immediate_close>, substate = bin_reading_packet, registered_in_libevent = false, event = {ev_active_next = { tqe_next = 0x0, tqe_prev = 0x19b88d0} , ev_next = {tqe_next = 0x0, tqe_prev = 0x4b8ef48} , ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev = 0x0} , min_heap_idx = -1}, ev_fd = 83, ev_base = 0x19b84a0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x7fbfe443bcd0} , ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = { ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x7fbfe443bcd0} , ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 22, ev_res = 0, ev_flags = 128, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = { tv_sec = 0, tv_usec = 0} , ev_callback = 0x406fa0 <event_handler>, ev_arg = 0x407a8e8}, ev_flags = 22, which = 2, rbuf = 0xe4342e0 "\201A", rcurr = 0xe4342e0 "\201A", rsize = 2048, rbytes = 0, wbuf = 0x407b3a0 "\200A", wcurr = 0x407b3a0 "\200A", wsize = 2048, wbytes = 78272, write_and_go = 0x412760 <conn_ship_log>, write_and_free = 0x0, ritem = 0xe4342f8 "\201D", rlbytes = 0, item = 0x0, store_op = 0, sbytes = 0, iov = 0x407c2b0, iovsize = 400, iovused = 0, msglist = 0x407dbc0, msgsize = 10, msgused = 1, msgcurr = 0, msgbytes = 0, ilist = 0x407bbb0, isize = 200, icurr = 0x407bbb0, ileft = 0, suffixlist = 0x407c200, suffixsize = 20, suffixcurr = 0x407c200, suffixleft = 0, protocol = binary_prot, transport = tcp_transport, request_id = 0, request_addr = { ss_family = 0, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>} , request_addr_size = 0, hdrbuf = 0x0, hdrsize = 0, noreply = false, refcount = 1 '\001', dynamic_buffer = {buffer = 0x0, size = 2048, offset = 32} , engine_storage = 0x8ef9370, ascii_cmd = 0x0, binary_header = {request = { magic = 129 '\201', opcode = 65 'A', keylen = 0, extlen = 0 '\000', datatype = 0 '\000', vbucket = 0, bodylen = 0, opaque = 644349952, cas = 0} , bytes = "\201A", '\000' <repeats 12 times>, "h&\000\000\000\000\000\000\000"}, cas = 0, cmd = 65, opaque = 644349952, keylen = 0, list_state = 0, next = 0x0, thread = 0x19b5fc0, aiostat = ENGINE_SUCCESS, ewouldblock = true, tap_nack_mode = true, tap_iterator = 0x7fbff0061789 <bucket_tap_iterator_shim>} The refcount is 1 here meaning that the tap thread shouldn't try to reference this object at all (that's the core's reference)...
        Hide
        trond Trond Norbye added a comment -

        Please preserve the core file and the binaries you used so that I can dig more into the file later on today...

        Show
        trond Trond Norbye added a comment - Please preserve the core file and the binaries you used so that I can dig more into the file later on today...
        Hide
        sharon Sharon Barr (Inactive) added a comment -

        looks like i reproduce it (according to Trond's analysis of the core)
        Loaded 1 node with 1M items.
        While loading, rebalance a second node. rebalance was successful, all 1M items were loaded.
        add a third node - reblance failed with core dump:

        Core can be found at 10.2.1.12/tmp/core.14170

        [root@localhost tmp]# /opt/membase/bin/analyze_core /tmp/core.14170

        /tmp/core.14170.log contains initial debug information to try to identify
        the problem. The log may not contain all information we may need,
        so it is recommended that you preserve the corefile and the binary
        for a more deep analysis. If the program use any shared libraries
        (such as memcached engines) you should also preserve them.

        Please send /tmp/core.14170.log to support@membase.com

        [root@localhost tmp]# ls
        atop.d core.14170 core.14170.log hsperfdata_root mapping-root
        [root@localhost tmp]# cat core.14170.log
        Basic crash dump analysis of /tmp/core.14170.

        Please send the file to support@membase.com

        --------------------------------------------------------------------------------
        File information:
        -rwxr-xr-x 1 bin bin 392304 May 6 19:47 /opt/membase/./bin/memcached
        rw------ 1 membase membase 2782642176 May 7 09:51 /tmp/core.14170
        1ef0c463296c3cf36d535a39b484a177 /opt/membase/./bin/memcached
        5d0b5364041351d84cf4a27420bf7f93 /tmp/core.14170
        memcached 1.4.4_459_g6e709ec
        memcached 1.4.4_459_g6e709ec
        VERSION 1.4.4_459_g6e709ec
        --------------------------------------------------------------------------------
        Core file callstacks:
        GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
        Copyright (C) 2009 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law. Type "show copying"
        and "show warranty" for details.
        This GDB was configured as "x86_64-redhat-linux-gnu".
        For bug reporting instructions, please see:
        <http://www.gnu.org/software/gdb/bugs/>...
        Reading symbols from /opt/membase/bin/memcached...done.
        [New Thread 14180]
        [New Thread 14179]
        [New Thread 14178]
        [New Thread 14177]
        [New Thread 14176]
        [New Thread 14175]
        [New Thread 14174]
        [New Thread 14173]
        [New Thread 14172]
        [New Thread 14171]
        [New Thread 14170]
        Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done.
        Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0
        Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done.
        Loaded symbols for /opt/membase/lib/libevent-2.0.so.5
        Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libdl.so.2
        Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libm.so.6
        Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/librt.so.1
        Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libpthread.so.0
        Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libc.so.6
        Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/ld-linux-x86-64.so.2
        Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done.
        Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so
        Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done.
        Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so
        Reading symbols from /opt/membase/lib/memcached/ep.so...done.
        Loaded symbols for /opt/membase/lib/memcached/ep.so
        Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
        Loaded symbols for /usr/lib64/libstdc++.so.6
        Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libgcc_s.so.1
        Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'.
        Program terminated with signal 6, Aborted.
        #0 0x000000310f230265 in raise () from /lib64/libc.so.6

        Thread 12 (Thread 14170):
        #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0c3b10, tv=<value optimized out>) at epoll.c:404
        #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0c3b10, flags=0) at event.c:1558
        #3 0x0000000000409974 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7459

        Thread 11 (Thread 14171):
        #0 0x000000310f2c5f3b in read () from /lib64/libc.so.6
        #1 0x000000310f26cc07 in _IO_new_file_underflow () from /lib64/libc.so.6
        #2 0x000000310f26d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6
        #3 0x000000310f268e8b in getc () from /lib64/libc.so.6
        #4 0x00002b5eb1ee4855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19
        #5 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #6 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 10 (Thread 14172):
        #0 0x000000310f29a0b1 in nanosleep () from /lib64/libc.so.6
        #1 0x000000310f299ed4 in sleep () from /lib64/libc.so.6
        #2 0x0000000000414f17 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:220
        #3 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #4 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 9 (Thread 14173):
        #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0cf100, tv=<value optimized out>) at epoll.c:404
        #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0cf100, flags=0) at event.c:1558
        #3 0x0000000000413394 in worker_libevent (arg=0xb0ceba0) at daemon/thread.c:304
        #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 8 (Thread 14174):
        #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0cf9a0, tv=<value optimized out>) at epoll.c:404
        #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0cf9a0, flags=0) at event.c:1558
        #3 0x0000000000413394 in worker_libevent (arg=0xb0ceca0) at daemon/thread.c:304
        #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 7 (Thread 14175):
        #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0d0240, tv=<value optimized out>) at epoll.c:404
        #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0d0240, flags=0) at event.c:1558
        #3 0x0000000000413394 in worker_libevent (arg=0xb0ceda0) at daemon/thread.c:304
        #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 6 (Thread 14176):
        #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0d0ae0, tv=<value optimized out>) at epoll.c:404
        #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0d0ae0, flags=0) at event.c:1558
        #3 0x0000000000413394 in worker_libevent (arg=0xb0ceea0) at daemon/thread.c:304
        #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 5 (Thread 14177):
        #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
        #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0d1480, tv=<value optimized out>) at epoll.c:404
        #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0d1480, flags=0) at event.c:1558
        #3 0x0000000000413394 in worker_libevent (arg=0xb0cefa0) at daemon/thread.c:304
        #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 4 (Thread 14178):
        #0 0x000000310fe0ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaab40dc1be in SyncObject::wait (this=0x2aaab02d2570) at syncobject.hh:31
        #2 0x00002aaab40d846d in Dispatcher::run (this=0x2aaab02d2560) at dispatcher.cc:85
        #3 0x00002aaab40d8cac in launch_dispatcher_thread (arg=0x2aaab02d2560) at dispatcher.cc:28
        #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 3 (Thread 14179):
        #0 0x000000310fe0dd7b in lseek64 () from /lib64/libpthread.so.0
        #1 0x00002aaab416f2bf in seekAndRead (id=0xb162490, offset=323301376, pBuf=0x2aaaac218f70, cnt=1024) at embedded/sqlite3.c:25237
        #2 0x00002aaab416f363 in unixRead (id=0xb162490, pBuf=0x2aaaac218f70, amt=1024, offset=323301376) at embedded/sqlite3.c:25281
        #3 0x00002aaab4166c0b in sqlite3OsRead (id=0xb162490, pBuf=0x2aaaac218f70, amt=1024, offset=323301376) at embedded/sqlite3.c:13402
        #4 0x00002aaab4175e0e in readDbPage (pPg=0x2aaaac218f28) at embedded/sqlite3.c:36959
        #5 0x00002aaab4177ea8 in sqlite3PagerAcquire (pPager=0xb162348, pgno=315725, ppPage=0x4648c280, noContent=0) at embedded/sqlite3.c:39100
        #6 0x00002aaab417ef60 in btreeGetPage (pBt=0xb163bc8, pgno=315725, ppPage=0x4648c3c8, noContent=0) at embedded/sqlite3.c:46084
        #7 0x00002aaab4183a02 in allocateBtreePage (pBt=0xb163bc8, ppPage=0x4648c3c8, pPgno=0x4648c3c4, nearby=0, exact=0 '\000') at embedded/sqlite3.c:49433
        #8 0x00002aaab41843c3 in fillInCell (pPage=0x2aaac05e0850, pCell=0xb16d878 "\220\003\202Q\t\002\001,\001\001\003\237P\002\255", pKey=0x0, nKey=337, pData=0x2aab023aa968, nData=0, nZero=0, pnSize=0x4648c470) at embedded/sqlite3.c:49792
        #9 0x00002aaab4186b37 in sqlite3BtreeInsert (pCur=0x2aab0096d958, pKey=0x0, nKey=337, pData=0x2aab023aa968, nData=2051, nZero=0, appendBias=0, seekResult=0) at embedded/sqlite3.c:51201
        #10 0x00002aaab41987fc in sqlite3VdbeExec (p=0xbd474a8) at embedded/sqlite3.c:63371
        #11 0x00002aaab4191016 in sqlite3Step (p=0xbd474a8) at embedded/sqlite3.c:57947
        #12 0x00002aaab41911d8 in sqlite3_step (pStmt=0xbd474a8) at embedded/sqlite3.c:58011
        #13 0x00002aaab4159f76 in PreparedStatement::execute (this=0xbd44420) at sqlite-pst.cc:73
        #14 0x00002aaab4157b3b in StrategicSqlite3::update (this=0xb153420, itm=..., vb_version=0, cb=...) at sqlite-kvstore.cc:74
        #15 0x00002aaab4157ea5 in StrategicSqlite3::set (this=0xb153420, itm=..., vb_version=0, cb=...) at sqlite-kvstore.cc:110
        #16 0x00002aaab40dfc0e in EventuallyPersistentStore::flushOneDelOrSet (this=0x2aaab0000c50, qi=..., rejectQueue=0x2aab0282f510) at ep.cc:1828
        #17 0x00002aaab40dffd7 in EventuallyPersistentStore::flushOne (this=0x2aaab0000c50, q=0x2aaab0000d70, rejectQueue=0x2aab0282f510) at ep.cc:1871
        #18 0x00002aaab40e0191 in EventuallyPersistentStore::flushSome (this=0x2aaab0000c50, q=0x2aaab0000d70, rejectQueue=0x2aab0282f510) at ep.cc:1483
        #19 0x00002aaab4128f7e in Flusher::doFlush (this=0x2aaab108eb40) at flusher.cc:240
        #20 0x00002aaab41293c0 in Flusher::step (this=0x2aaab108eb40, d=..., tid=...) at flusher.cc:154
        #21 0x00002aaab4129859 in FlusherStepper::callback (this=0xc0a3340, d=..., t=...) at flusher.cc:23
        #22 0x00002aaab40d9bf0 in Task::run (this=0xc0a1c90, d=..., t=...) at dispatcher.hh:139
        #23 0x00002aaab40d874c in Dispatcher::run (this=0x2aaab01520c0) at dispatcher.cc:119
        #24 0x00002aaab40d8cac in launch_dispatcher_thread (arg=0x2aaab01520c0) at dispatcher.cc:28
        #25 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #26 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 2 (Thread 14180):
        #0 0x000000310fe0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00002aaab40db0fc in SyncObject::wait (this=0x2aaab108da10, tv=...) at syncobject.hh:42
        #2 0x00002aaab40d7651 in IdleTask::run (this=0x2aaab108ea90, d=...) at dispatcher.cc:244
        #3 0x00002aaab40d874c in Dispatcher::run (this=0x2aaab108da00) at dispatcher.cc:119
        #4 0x00002aaab40d8cac in launch_dispatcher_thread (arg=0x2aaab108da00) at dispatcher.cc:28
        #5 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #6 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

        Thread 1 (Thread 14181):
        #0 0x000000310f230265 in raise () from /lib64/libc.so.6
        #1 0x000000310f231d10 in abort () from /lib64/libc.so.6
        #2 0x000000310f2296e6 in __assert_fail () from /lib64/libc.so.6
        #3 0x00002aaaaaab1a8d in bucket_engine_release_cookie (cookie=0x2aab0efde648) at bucket_engine.c:1976
        #4 0x00002aaab414f8bc in TapConnMap::notifyIOThreadMain (this=0xb14abc0) at tapconnmap.cc:441
        #5 0x00002aaab40fd418 in EventuallyPersistentEngine::notifyTapIoThread (this=0xb14a620) at ep_engine.cc:3568
        #6 0x00002aaab41029aa in EvpNotifyTapIo (arg=0xb14a620) at ep_engine.cc:882
        #7 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
        #8 0x000000310f2d3c2d in clone () from /lib64/libc.so.6
        --------------------------------------------------------------------------------
        Module information:
        /opt/membase/lib/memcached/libmemcached_utilities.so.0:
        lrwxrwxrwx 1 bin bin 31 May 7 09:46 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
        c7fc13e0591be3155fea1efcd88c0430 /opt/membase/lib/memcached/libmemcached_utilities.so.0
        /opt/membase/lib/libevent-2.0.so.5:
        lrwxrwxrwx 1 bin bin 21 May 7 09:46 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
        62b1f1fede447a7642ca0f09d84eebe6 /opt/membase/lib/libevent-2.0.so.5
        /lib64/libdl.so.2:
        lrwxrwxrwx 1 root root 12 Nov 12 18:27 /lib64/libdl.so.2 -> libdl-2.5.so
        ca2f62a87e65c4a7c453c5f17007c97e /lib64/libdl.so.2
        /lib64/libm.so.6:
        lrwxrwxrwx 1 root root 11 Nov 12 18:27 /lib64/libm.so.6 -> libm-2.5.so
        96b2756d90a6a8cc813b4a4831eb0e59 /lib64/libm.so.6
        /lib64/librt.so.1:
        lrwxrwxrwx 1 root root 12 Nov 12 18:27 /lib64/librt.so.1 -> librt-2.5.so
        bd2393c8e729efe0e7210141a9e8192b /lib64/librt.so.1
        /lib64/libpthread.so.0:
        lrwxrwxrwx 1 root root 17 Nov 12 18:27 /lib64/libpthread.so.0 -> libpthread-2.5.so
        fa9fe0ece649c45833ed36d320980e06 /lib64/libpthread.so.0
        /lib64/libc.so.6:
        lrwxrwxrwx 1 root root 11 Nov 12 18:27 /lib64/libc.so.6 -> libc-2.5.so
        4e4dda03ce8f1d41088b55a606736078 /lib64/libc.so.6
        /lib64/ld-linux-x86-64.so.2:
        lrwxrwxrwx 1 root root 9 Nov 12 18:27 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so
        d5bc07ec4dbf877d8abaa963ab52342a /lib64/ld-linux-x86-64.so.2
        /opt/membase/lib/memcached/stdin_term_handler.so:
        -rwxr-xr-x 1 bin bin 15322 May 6 19:47 /opt/membase/lib/memcached/stdin_term_handler.so
        2b46627c47610208b90a34190ab97bb8 /opt/membase/lib/memcached/stdin_term_handler.so
        /opt/membase/lib/memcached/bucket_engine.so:
        lrwxrwxrwx 1 bin bin 22 May 7 09:46 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
        3172eba599233ba7534d598e65afa23b /opt/membase/lib/memcached/bucket_engine.so
        /opt/membase/lib/memcached/ep.so:
        lrwxrwxrwx 1 bin bin 11 May 7 09:46 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0
        1.6.5.3_237_g6656d4b
        EP engine v1.6.5.3_237_g6656d4b
        1c1c829c48566e1f3d0ca1a1008dd56a /opt/membase/lib/memcached/ep.so
        /usr/lib64/libstdc++.so.6:
        lrwxrwxrwx 1 root root 18 Nov 12 18:27 /usr/lib64/libstdc+.so.6 -> libstdc+.so.6.0.8
        f899654bffff02d2d02520cd909678ac /usr/lib64/libstdc++.so.6
        /lib64/libgcc_s.so.1:
        lrwxrwxrwx 1 root root 28 Nov 12 18:27 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1
        ea8989f21880353a78e5a0b0291b4efe /lib64/libgcc_s.so.1
        [root@localhost tmp]#

        Show
        sharon Sharon Barr (Inactive) added a comment - looks like i reproduce it (according to Trond's analysis of the core) Loaded 1 node with 1M items. While loading, rebalance a second node. rebalance was successful, all 1M items were loaded. add a third node - reblance failed with core dump: Core can be found at 10.2.1.12/tmp/core.14170 [root@localhost tmp] # /opt/membase/bin/analyze_core /tmp/core.14170 /tmp/core.14170.log contains initial debug information to try to identify the problem. The log may not contain all information we may need, so it is recommended that you preserve the corefile and the binary for a more deep analysis. If the program use any shared libraries (such as memcached engines) you should also preserve them. Please send /tmp/core.14170.log to support@membase.com [root@localhost tmp] # ls atop.d core.14170 core.14170.log hsperfdata_root mapping-root [root@localhost tmp] # cat core.14170.log Basic crash dump analysis of /tmp/core.14170. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 392304 May 6 19:47 /opt/membase/./bin/memcached rw ------ 1 membase membase 2782642176 May 7 09:51 /tmp/core.14170 1ef0c463296c3cf36d535a39b484a177 /opt/membase/./bin/memcached 5d0b5364041351d84cf4a27420bf7f93 /tmp/core.14170 memcached 1.4.4_459_g6e709ec memcached 1.4.4_459_g6e709ec VERSION 1.4.4_459_g6e709ec -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: < http://www.gnu.org/software/gdb/bugs/ >... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 14180] [New Thread 14179] [New Thread 14178] [New Thread 14177] [New Thread 14176] [New Thread 14175] [New Thread 14174] [New Thread 14173] [New Thread 14172] [New Thread 14171] [New Thread 14170] Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x000000310f230265 in raise () from /lib64/libc.so.6 Thread 12 (Thread 14170): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0c3b10, tv=<value optimized out>) at epoll.c:404 #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0c3b10, flags=0) at event.c:1558 #3 0x0000000000409974 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7459 Thread 11 (Thread 14171): #0 0x000000310f2c5f3b in read () from /lib64/libc.so.6 #1 0x000000310f26cc07 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x000000310f26d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x000000310f268e8b in getc () from /lib64/libc.so.6 #4 0x00002b5eb1ee4855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19 #5 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #6 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 10 (Thread 14172): #0 0x000000310f29a0b1 in nanosleep () from /lib64/libc.so.6 #1 0x000000310f299ed4 in sleep () from /lib64/libc.so.6 #2 0x0000000000414f17 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:220 #3 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 9 (Thread 14173): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0cf100, tv=<value optimized out>) at epoll.c:404 #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0cf100, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xb0ceba0) at daemon/thread.c:304 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 8 (Thread 14174): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0cf9a0, tv=<value optimized out>) at epoll.c:404 #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0cf9a0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xb0ceca0) at daemon/thread.c:304 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 7 (Thread 14175): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0d0240, tv=<value optimized out>) at epoll.c:404 #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0d0240, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xb0ceda0) at daemon/thread.c:304 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 6 (Thread 14176): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0d0ae0, tv=<value optimized out>) at epoll.c:404 #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0d0ae0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xb0ceea0) at daemon/thread.c:304 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 5 (Thread 14177): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b5eb1cb1c28 in epoll_dispatch (base=0xb0d1480, tv=<value optimized out>) at epoll.c:404 #2 0x00002b5eb1ca0a4c in event_base_loop (base=0xb0d1480, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xb0cefa0) at daemon/thread.c:304 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 4 (Thread 14178): #0 0x000000310fe0ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaab40dc1be in SyncObject::wait (this=0x2aaab02d2570) at syncobject.hh:31 #2 0x00002aaab40d846d in Dispatcher::run (this=0x2aaab02d2560) at dispatcher.cc:85 #3 0x00002aaab40d8cac in launch_dispatcher_thread (arg=0x2aaab02d2560) at dispatcher.cc:28 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 3 (Thread 14179): #0 0x000000310fe0dd7b in lseek64 () from /lib64/libpthread.so.0 #1 0x00002aaab416f2bf in seekAndRead (id=0xb162490, offset=323301376, pBuf=0x2aaaac218f70, cnt=1024) at embedded/sqlite3.c:25237 #2 0x00002aaab416f363 in unixRead (id=0xb162490, pBuf=0x2aaaac218f70, amt=1024, offset=323301376) at embedded/sqlite3.c:25281 #3 0x00002aaab4166c0b in sqlite3OsRead (id=0xb162490, pBuf=0x2aaaac218f70, amt=1024, offset=323301376) at embedded/sqlite3.c:13402 #4 0x00002aaab4175e0e in readDbPage (pPg=0x2aaaac218f28) at embedded/sqlite3.c:36959 #5 0x00002aaab4177ea8 in sqlite3PagerAcquire (pPager=0xb162348, pgno=315725, ppPage=0x4648c280, noContent=0) at embedded/sqlite3.c:39100 #6 0x00002aaab417ef60 in btreeGetPage (pBt=0xb163bc8, pgno=315725, ppPage=0x4648c3c8, noContent=0) at embedded/sqlite3.c:46084 #7 0x00002aaab4183a02 in allocateBtreePage (pBt=0xb163bc8, ppPage=0x4648c3c8, pPgno=0x4648c3c4, nearby=0, exact=0 '\000') at embedded/sqlite3.c:49433 #8 0x00002aaab41843c3 in fillInCell (pPage=0x2aaac05e0850, pCell=0xb16d878 "\220\003\202Q\t\002\001,\001\001\003\237P\002\255", pKey=0x0, nKey=337, pData=0x2aab023aa968, nData=0, nZero=0, pnSize=0x4648c470) at embedded/sqlite3.c:49792 #9 0x00002aaab4186b37 in sqlite3BtreeInsert (pCur=0x2aab0096d958, pKey=0x0, nKey=337, pData=0x2aab023aa968, nData=2051, nZero=0, appendBias=0, seekResult=0) at embedded/sqlite3.c:51201 #10 0x00002aaab41987fc in sqlite3VdbeExec (p=0xbd474a8) at embedded/sqlite3.c:63371 #11 0x00002aaab4191016 in sqlite3Step (p=0xbd474a8) at embedded/sqlite3.c:57947 #12 0x00002aaab41911d8 in sqlite3_step (pStmt=0xbd474a8) at embedded/sqlite3.c:58011 #13 0x00002aaab4159f76 in PreparedStatement::execute (this=0xbd44420) at sqlite-pst.cc:73 #14 0x00002aaab4157b3b in StrategicSqlite3::update (this=0xb153420, itm=..., vb_version=0, cb=...) at sqlite-kvstore.cc:74 #15 0x00002aaab4157ea5 in StrategicSqlite3::set (this=0xb153420, itm=..., vb_version=0, cb=...) at sqlite-kvstore.cc:110 #16 0x00002aaab40dfc0e in EventuallyPersistentStore::flushOneDelOrSet (this=0x2aaab0000c50, qi=..., rejectQueue=0x2aab0282f510) at ep.cc:1828 #17 0x00002aaab40dffd7 in EventuallyPersistentStore::flushOne (this=0x2aaab0000c50, q=0x2aaab0000d70, rejectQueue=0x2aab0282f510) at ep.cc:1871 #18 0x00002aaab40e0191 in EventuallyPersistentStore::flushSome (this=0x2aaab0000c50, q=0x2aaab0000d70, rejectQueue=0x2aab0282f510) at ep.cc:1483 #19 0x00002aaab4128f7e in Flusher::doFlush (this=0x2aaab108eb40) at flusher.cc:240 #20 0x00002aaab41293c0 in Flusher::step (this=0x2aaab108eb40, d=..., tid=...) at flusher.cc:154 #21 0x00002aaab4129859 in FlusherStepper::callback (this=0xc0a3340, d=..., t=...) at flusher.cc:23 #22 0x00002aaab40d9bf0 in Task::run (this=0xc0a1c90, d=..., t=...) at dispatcher.hh:139 #23 0x00002aaab40d874c in Dispatcher::run (this=0x2aaab01520c0) at dispatcher.cc:119 #24 0x00002aaab40d8cac in launch_dispatcher_thread (arg=0x2aaab01520c0) at dispatcher.cc:28 #25 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #26 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 2 (Thread 14180): #0 0x000000310fe0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaab40db0fc in SyncObject::wait (this=0x2aaab108da10, tv=...) at syncobject.hh:42 #2 0x00002aaab40d7651 in IdleTask::run (this=0x2aaab108ea90, d=...) at dispatcher.cc:244 #3 0x00002aaab40d874c in Dispatcher::run (this=0x2aaab108da00) at dispatcher.cc:119 #4 0x00002aaab40d8cac in launch_dispatcher_thread (arg=0x2aaab108da00) at dispatcher.cc:28 #5 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #6 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 1 (Thread 14181): #0 0x000000310f230265 in raise () from /lib64/libc.so.6 #1 0x000000310f231d10 in abort () from /lib64/libc.so.6 #2 0x000000310f2296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00002aaaaaab1a8d in bucket_engine_release_cookie (cookie=0x2aab0efde648) at bucket_engine.c:1976 #4 0x00002aaab414f8bc in TapConnMap::notifyIOThreadMain (this=0xb14abc0) at tapconnmap.cc:441 #5 0x00002aaab40fd418 in EventuallyPersistentEngine::notifyTapIoThread (this=0xb14a620) at ep_engine.cc:3568 #6 0x00002aaab41029aa in EvpNotifyTapIo (arg=0xb14a620) at ep_engine.cc:882 #7 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #8 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 May 7 09:46 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 c7fc13e0591be3155fea1efcd88c0430 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 May 7 09:46 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 62b1f1fede447a7642ca0f09d84eebe6 /opt/membase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 12 Nov 12 18:27 /lib64/libdl.so.2 -> libdl-2.5.so ca2f62a87e65c4a7c453c5f17007c97e /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 11 Nov 12 18:27 /lib64/libm.so.6 -> libm-2.5.so 96b2756d90a6a8cc813b4a4831eb0e59 /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 12 Nov 12 18:27 /lib64/librt.so.1 -> librt-2.5.so bd2393c8e729efe0e7210141a9e8192b /lib64/librt.so.1 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 17 Nov 12 18:27 /lib64/libpthread.so.0 -> libpthread-2.5.so fa9fe0ece649c45833ed36d320980e06 /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 11 Nov 12 18:27 /lib64/libc.so.6 -> libc-2.5.so 4e4dda03ce8f1d41088b55a606736078 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 9 Nov 12 18:27 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so d5bc07ec4dbf877d8abaa963ab52342a /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15322 May 6 19:47 /opt/membase/lib/memcached/stdin_term_handler.so 2b46627c47610208b90a34190ab97bb8 /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 May 7 09:46 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 3172eba599233ba7534d598e65afa23b /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 May 7 09:46 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 1.6.5.3_237_g6656d4b EP engine v1.6.5.3_237_g6656d4b 1c1c829c48566e1f3d0ca1a1008dd56a /opt/membase/lib/memcached/ep.so /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 18 Nov 12 18:27 /usr/lib64/libstdc+ .so.6 -> libstdc +.so.6.0.8 f899654bffff02d2d02520cd909678ac /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Nov 12 18:27 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1 ea8989f21880353a78e5a0b0291b4efe /lib64/libgcc_s.so.1 [root@localhost tmp] #
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        seeing this crash again on a build which should have more debugging info based on Trond's recent changes.

        Thread 1 (Thread 23034):
        #0 0xb7894430 in __kernel_vsyscall ()
        #1 0xb76e34d1 in raise () from /lib/tls/i686/cmov/libc.so.6
        #2 0xb76e6932 in abort () from /lib/tls/i686/cmov/libc.so.6
        #3 0xb76dc648 in __assert_fail () from /lib/tls/i686/cmov/libc.so.6
        #4 0xb66ae464 in bucket_engine_release_cookie (cookie=0xb1e0688) at bucket_engine.c:1989
        #5 0xb3df0592 in TapConnection::releaseReference (this=0xb04e198, force=false) at tapconnection.cc:31
        #6 0xb3e00d90 in TapConnectionReaperCallback (this=0x95d45bc) at tapconnmap.cc:21
        #7 TapConnMap::shutdownAllTapConnections (this=0x95d45bc) at tapconnmap.cc:341
        #8 0xb3dbc66c in EventuallyPersistentEngine::destroy (this=0x95d42e0, force=true) at ep_engine.cc:1438
        #9 0xb3dc5597 in EvpDestroy (handle=0x95d42e0, force=<value optimized out>) at ep_engine.cc:96
        #10 0xb66adbac in engine_shutdown_thread (arg=0x9580f48) at bucket_engine.c:1096
        #11 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
        #12 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6

        Show
        farshid Farshid Ghods (Inactive) added a comment - seeing this crash again on a build which should have more debugging info based on Trond's recent changes. Thread 1 (Thread 23034): #0 0xb7894430 in __kernel_vsyscall () #1 0xb76e34d1 in raise () from /lib/tls/i686/cmov/libc.so.6 #2 0xb76e6932 in abort () from /lib/tls/i686/cmov/libc.so.6 #3 0xb76dc648 in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 #4 0xb66ae464 in bucket_engine_release_cookie (cookie=0xb1e0688) at bucket_engine.c:1989 #5 0xb3df0592 in TapConnection::releaseReference (this=0xb04e198, force=false) at tapconnection.cc:31 #6 0xb3e00d90 in TapConnectionReaperCallback (this=0x95d45bc) at tapconnmap.cc:21 #7 TapConnMap::shutdownAllTapConnections (this=0x95d45bc) at tapconnmap.cc:341 #8 0xb3dbc66c in EventuallyPersistentEngine::destroy (this=0x95d42e0, force=true) at ep_engine.cc:1438 #9 0xb3dc5597 in EvpDestroy (handle=0x95d42e0, force=<value optimized out>) at ep_engine.cc:96 #10 0xb66adbac in engine_shutdown_thread (arg=0x9580f48) at bucket_engine.c:1096 #11 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #12 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        attached the core

        Show
        farshid Farshid Ghods (Inactive) added a comment - attached the core
        Hide
        farshid Farshid Ghods (Inactive) added a comment - - edited

        attached the diag file which is 9.1 MB

        the crash happened @ 7:15

        rw------ 1 membase membase 489455616 2011-05-08 07:15 /tmp/core.memcached.20386

        Show
        farshid Farshid Ghods (Inactive) added a comment - - edited attached the diag file which is 9.1 MB the crash happened @ 7:15 rw ------ 1 membase membase 489455616 2011-05-08 07:15 /tmp/core.memcached.20386
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Integrated in github-bucket-engine #16 (See http://qa.hq.northscale.net/job/github-bucket-engine/16/)
        MB-3764 Don't release the engine handle if the bucket is in the wrong state

        Chiyoung Seo :
        Files :

        • bucket_engine.c
        Show
        farshid Farshid Ghods (Inactive) added a comment - Integrated in github-bucket-engine #16 (See http://qa.hq.northscale.net/job/github-bucket-engine/16/ ) MB-3764 Don't release the engine handle if the bucket is in the wrong state Chiyoung Seo : Files : bucket_engine.c

          People

          • Assignee:
            trond Trond Norbye
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes