Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-3756

deleting database directory makes ep-engine 'hang' in TransactionContext::commit

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Trivial
    • 1.7.1
    • None
    • couchbase-bucket, ns_server
    • Security Level: Public
    • None

    Description

      I've abruptly stopped service and did make dataclean which deleted data directory. But I'm seeing memcacheds being stuck. Here's backtrace:

      oaded symbols for /lib/libgcc_s.so.1
      0xffffe430 in __kernel_vsyscall ()
      (gdb) thread apply all bt

      Thread 5 (Thread 0xf6ca5b70 (LWP 21388)):
      #0 0xffffe430 in __kernel_vsyscall ()
      #1 0xf75a03f6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
      #2 0xf75a0220 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
      #3 0x08062317 in check_isasl_db_thread (arg=0x806afd4) at daemon/isasl.c:220
      #4 0xf76b8c39 in start_thread (arg=0xf6ca5b70) at pthread_create.c:304
      #5 0xf75d591e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

      Thread 4 (Thread 0xf381db70 (LWP 21400)):
      #0 0xffffe430 in __kernel_vsyscall ()
      #1 0xf76bd20a in __pthread_cond_wait (cond=0x97fcce8, mutex=0x97fcccc) at pthread_cond_wait.c:153
      #2 0xf39a70d0 in SyncObject::wait (this=0x97fccc8) at syncobject.hh:31
      #3 0xf39a5ce3 in Dispatcher::run (this=0x97fccc0) at dispatcher.cc:85
      #4 0xf39a5853 in launch_dispatcher_thread (arg=0x97fccc0) at dispatcher.cc:28
      #5 0xf76b8c39 in start_thread (arg=0xf381db70) at pthread_create.c:304
      #6 0xf75d591e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

      Thread 3 (Thread 0xf301cb70 (LWP 21401)):
      #0 0xffffe430 in __kernel_vsyscall ()
      #1 0xf75a03f6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
      #2 0xf75a0220 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
      #3 0xf39b274b in TransactionContext::commit (this=0x8b875e4) at ep.cc:2114
      #4 0xf39b01e5 in EventuallyPersistentStore::flushSome (this=0x8b87500, q=0x8b875b0, rejectQueue=0xb2853c0) at ep.cc:1491
      #5 0xf39f36da in Flusher::doFlush (this=0xa362810) at flusher.cc:240
      #6 0xf39f34dc in Flusher::completeFlush (this=0xa362810) at flusher.cc:202
      #7 0xf39f3284 in Flusher::step (this=0xa362810, d=..., tid=std::tr1::shared_ptr (count 5) 0xabfec38) at flusher.cc:174
      #8 0xf39f28d1 in FlusherStepper::callback (this=0xac5dbb0, d=..., t=std::tr1::shared_ptr (count 5) 0xabfec38) at flusher.cc:23
      #9 0xf39a76fa in Task::run (this=0xabfec38, d=..., t=std::tr1::shared_ptr (count 5) 0xabfec38) at dispatcher.hh:139
      #10 0xf39a5f6c in Dispatcher::run (this=0x92b2378) at dispatcher.cc:119
      #11 0xf39a5853 in launch_dispatcher_thread (arg=0x92b2378) at dispatcher.cc:28
      #12 0xf76b8c39 in start_thread (arg=0xf301cb70) at pthread_create.c:304
      #13 0xf75d591e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
      --Type <return> to continue, or q <return> to quit--

      Thread 2 (Thread 0xf281bb70 (LWP 21402)):
      #0 0xffffe430 in __kernel_vsyscall ()
      #1 0xf76bd703 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
      at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
      #2 0xf39a71de in SyncObject::wait (this=0xa361c60, tv=...) at syncobject.hh:42
      #3 0xf39a6d77 in IdleTask::run (this=0xa362798, d=...) at dispatcher.cc:244
      #4 0xf39a5f6c in Dispatcher::run (this=0xa361c58) at dispatcher.cc:119
      #5 0xf39a5853 in launch_dispatcher_thread (arg=0xa361c58) at dispatcher.cc:28
      #6 0xf76b8c39 in start_thread (arg=0xf281bb70) at pthread_create.c:304
      #7 0xf75d591e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

      Thread 1 (Thread 0xf74dc9a0 (LWP 21384)):
      #0 0xffffe430 in __kernel_vsyscall ()
      #1 0xf75a03f6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
      #2 0xf75cf61c in usleep (useconds=1000) at ../sysdeps/unix/sysv/linux/usleep.c:33
      #3 0xf39f2947 in Flusher::wait (this=0xa362810) at flusher.cc:33
      #4 0xf39aa781 in EventuallyPersistentStore::stopFlusher (this=0x8b87500) at ep.cc:447
      #5 0xf39aa4aa in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x8b87500, __in_chrg=<value optimized out>)
      at ep.cc:414
      #6 0xf39dac41 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x8936280, __in_chrg=<value optimized out>)
      at ep_engine.h:505
      #7 0xf39c6706 in EvpDestroy (handle=0x8936280, force=false) at ep_engine.cc:97
      #8 0xf76e3da9 in bucket_shutdown_engine (key=0x893b810, nkey=7, val=0x8905c40, nval=0, args=0x0) at bucket_engine.c:1027
      #9 0xf76b0de7 in genhash_iter (h=0x88bd4a8, iterfunc=0xf76e3d3e <bucket_shutdown_engine>, arg=0x0) at utilities/genhash.c:270
      #10 0xf76e3e7e in bucket_destroy (handle=0xf76e8220, force=false) at bucket_engine.c:1051
      #11 0x0805e1b1 in main (argc=17, argv=0xffa04d74) at daemon/memcached.c:7467
      (gdb)
      (gdb) thread 3
      [Switching to thread 3 (Thread 0xf301cb70 (LWP 21401))]#0 0xffffe430 in __kernel_vsyscall ()
      (gdb) fin
      Run till exit from #0 0xffffe430 in __kernel_vsyscall ()
      0xf75a03f6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
      82 ../sysdeps/unix/syscall-template.S: No such file or directory.
      in ../sysdeps/unix/syscall-template.S
      (gdb)
      Run till exit from #0 0xf75a03f6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
      __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:139
      139 ../sysdeps/unix/sysv/linux/sleep.c: No such file or directory.
      in ../sysdeps/unix/sysv/linux/sleep.c
      (gdb)
      Run till exit from #0 __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:139
      TransactionContext::commit (this=0x8b875e4) at ep.cc:2115
      2115 ++stats.commitFailed;
      Value returned is $1 = 0
      (gdb) s
      Atomic<unsigned int>::operator++ (this=0x893669c) at atomic.hh:130
      130 return ep_sync_add_and_fetch(&value, 1);
      (gdb) bt
      #0 Atomic<unsigned int>::operator++ (this=0x893669c) at atomic.hh:130
      #1 0xf39b275b in TransactionContext::commit (this=0x8b875e4) at ep.cc:2115
      #2 0xf39b01e5 in EventuallyPersistentStore::flushSome (this=0x8b87500, q=0x8b875b0, rejectQueue=0xb2853c0) at ep.cc:1491
      #3 0xf39f36da in Flusher::doFlush (this=0xa362810) at flusher.cc:240
      #4 0xf39f34dc in Flusher::completeFlush (this=0xa362810) at flusher.cc:202
      #5 0xf39f3284 in Flusher::step (this=0xa362810, d=..., tid=std::tr1::shared_ptr (count 5) 0xabfec38) at flusher.cc:174
      #6 0xf39f28d1 in FlusherStepper::callback (this=0xac5dbb0, d=..., t=std::tr1::shared_ptr (count 5) 0xabfec38) at flusher.cc:23
      #7 0xf39a76fa in Task::run (this=0xabfec38, d=..., t=std::tr1::shared_ptr (count 5) 0xabfec38) at dispatcher.hh:139
      #8 0xf39a5f6c in Dispatcher::run (this=0x92b2378) at dispatcher.cc:119
      #9 0xf39a5853 in launch_dispatcher_thread (arg=0x92b2378) at dispatcher.cc:28
      #10 0xf76b8c39 in start_thread (arg=0xf301cb70) at pthread_create.c:304
      #11 0xf75d591e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
      (gdb) fin
      Run till exit from #0 Atomic<unsigned int>::operator++ (this=0x893669c) at atomic.hh:130
      TransactionContext::commit (this=0x8b875e4) at ep.cc:2113
      2113 while (!underlying->commit()) {
      Value returned is $2 = 269
      (gdb) s
      StrategicSqlite3::commit (this=0x893bbf8) at sqlite-kvstore.hh:70
      70 if(intransaction)

      { (gdb) n 72 intransaction = (execute("commit") == -1); (gdb) n 75 return !intransaction; (gdb) p intransaction $3 = true (gdb) s 76 }

      (gdb)
      TransactionContext::commit (this=0x8b875e4) at ep.cc:2114
      2114 sleep(1);
      (gdb) q
      A debugging session is active.

      Inferior 1 [process 21384] will be detached.

      Quit anyway? (y or n) y

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Unassigned Unassigned
            alkondratenko Aleksey Kondratenko (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty