Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8146

memcached crashed with signal 11, Segmentation fault in HashTable::visit()

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      centos 64

      Description

      build 2.0.2-769
      on run
      http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/593/consoleFull
      I see several identical crashes

      andrey@baranouski:~/repository/testrunner$ python scripts/ssh.py -i centos-64-2.0-failover-tests.ini "ls -la /tmp"
      10.1.3.114
      total 2105288
      drwxrwxrwt 5 root root 4096 Apr 22 09:11 .
      drwxr-xr-x 25 root root 4096 Jan 29 10:56 ..
      rw------ 1 couchbase couchbase 2229407744 Apr 22 05:17 core.memcached.26483
      rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
      drwxrwxrwt 2 root root 4096 Jan 29 10:56 .font-unix
      srw-rw-rw- 1 root root 0 Jan 29 10:56 .gdm_socket
      drwxrwxrwt 2 root root 4096 Jan 29 10:56 .ICE-unix
      rrr- 1 root root 11 Jan 29 10:56 .X0-lock
      drwxrwxrwt 2 root root 4096 Jan 29 10:56 .X11-unix

      10.1.3.117
      total 125044
      drwxrwxrwt 5 root root 413696 Apr 22 09:11 .
      drwxr-xr-x 25 root root 4096 Feb 11 13:26 ..
      rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
      drwxrwxrwt 2 root root 4096 Feb 11 13:26 .font-unix
      srw-rw-rw- 1 root root 0 Feb 11 13:26 .gdm_socket
      drwxrwxrwt 2 root root 4096 Feb 11 13:26 .ICE-unix
      rrr- 1 root root 11 Feb 11 13:26 .X0-lock
      drwxrwxrwt 2 root root 4096 Feb 11 13:26 .X11-unix

      10.1.3.115
      total 2083176
      drwxrwxrwt 5 root root 401408 Apr 22 09:11 .
      drwxr-xr-x 25 root root 4096 Oct 25 16:14 ..
      rw------ 1 couchbase couchbase 2157531136 Apr 22 05:24 core.memcached.1265
      rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
      drwxrwxrwt 2 root root 4096 Oct 25 16:19 .font-unix
      srw-rw-rw- 1 root root 0 Oct 25 16:19 .gdm_socket
      drwxrwxrwt 2 root root 4096 Oct 25 16:14 .ICE-unix
      rrr- 1 root root 11 Oct 25 16:19 .X0-lock
      drwxrwxrwt 2 root root 4096 Oct 25 16:19 .X11-unix

      10.1.3.118
      total 3928388
      drwxrwxrwt 6 root root 413696 Apr 22 09:11 .
      drwxr-xr-x 25 root root 4096 Oct 25 16:15 ..
      rw------ 1 couchbase couchbase 2261405696 Apr 22 04:01 core.memcached.23290
      rw------ 1 couchbase couchbase 2153013248 Apr 22 04:32 core.memcached.30599
      rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
      drwxrwxrwt 2 root root 4096 Oct 25 16:20 .font-unix
      srw-rw-rw- 1 root root 0 Oct 25 16:20 .gdm_socket
      drwxrwxrwt 2 root root 4096 Oct 25 16:15 .ICE-unix
      drwx------ 2 root root 4096 May 15 2012 keyring-cqfvn9
      rrr- 1 root root 11 Oct 25 16:20 .X0-lock
      drwxrwxrwt 2 root root 4096 Oct 25 16:20 .X11-unix

      10.1.3.116
      total 125016
      drwxrwxrwt 5 root root 385024 Apr 22 09:11 .
      drwxr-xr-x 25 root root 4096 Oct 25 16:15 ..
      rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
      drwxrwxrwt 2 root root 4096 Oct 25 16:19 .font-unix
      srw-rw-rw- 1 root root 0 Oct 25 16:20 .gdm_socket
      drwxrwxrwt 2 root root 4096 Oct 25 16:15 .ICE-unix
      rrr- 1 root root 11 Oct 25 16:20 .X0-lock
      drwxrwxrwt 2 root root 4096 Oct 25 16:20 .X11-unix

      [root@localhost tmp]# gdb /opt/couchbase/bin/memcached core.memcached.30599
      GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
      Copyright (C) 2009 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-redhat-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/bin/memcached...done.
      [New Thread 30684]
      [New Thread 30685]
      [New Thread 30683]
      [New Thread 30682]
      [New Thread 30681]
      [New Thread 30634]
      [New Thread 30633]
      [New Thread 30632]
      [New Thread 30631]
      [New Thread 30630]
      [New Thread 30621]
      [New Thread 30620]
      [New Thread 30619]
      [New Thread 30618]
      [New Thread 30617]
      [New Thread 30616]
      [New Thread 30615]
      [New Thread 30614]
      [New Thread 30613]
      [New Thread 30612]
      [New Thread 30611]
      [New Thread 30610]
      [New Thread 30609]
      [New Thread 30599]
      Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
      Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
      Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
      Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libdl.so.2
      Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libm.so.6
      Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/librt.so.1
      Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
      Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
      [Thread debugging using libthread_db enabled]
      Loaded symbols for /lib64/libpthread.so.0
      Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libstdc++.so.6
      Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libgcc_s.so.1
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
      Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
      Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
      Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libicudata.so.44
      Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

      warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff6d350000
      Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
      Program terminated with signal 11, Segmentation fault.
      #0 0x00002aaaaaf32d11 in HashTable::visit (this=0x2aaac5cce408, visitor=...) at src/stored-value.cc:395
      395 src/stored-value.cc: No such file or directory.
      in src/stored-value.cc
      (gdb) t a a bt

      Thread 24 (Thread 0x2adfa3764220 (LWP 30599)):
      #0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c000, tv=<value optimized out>) at epoll.c:404
      #2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c000, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7919

      Thread 23 (Thread 30609):
      #0 0x00000038286c678b in read () from /lib64/libc.so.6
      #1 0x000000382866cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
      #2 0x000000382866d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
      #3 0x0000003828662804 in _IO_getline_info_internal () from /lib64/libc.so.6
      #4 0x00000038286616a9 in fgets () from /lib64/libc.so.6
      #5 0x00002adfa3765939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
      #6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #7 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 22 (Thread 30610):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x42e8040) at extensions/loggers/file_logger.c:368
      #2 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #3 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 21 (Thread 30611):
      #0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c500, tv=<value optimized out>) at epoll.c:404
      #2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c500, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0x42eb900) at daemon/thread.c:301
      #4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 20 (Thread 30612):
      #0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c280, tv=<value optimized out>) at epoll.c:404
      #2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c280, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0x42eb9f8) at daemon/thread.c:301
      #4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 19 (Thread 30613):
      #0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4cc80, tv=<value optimized out>) at epoll.c:404
      #2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4cc80, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0x42ebaf0) at daemon/thread.c:301
      #4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 18 (Thread 30614):
      #0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4ca00, tv=<value optimized out>) at epoll.c:404
      #2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4ca00, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0x42ebbe8) at daemon/thread.c:301
      #4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000038286d44bd in clone () from /lib64/libc.so.6

      --Type <return> to continue, or q <return> to quit--
      Thread 17 (Thread 30615):
      #0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c780, tv=<value optimized out>) at epoll.c:404
      #2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c780, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000414574 in worker_libevent (arg=0x42ebce0) at daemon/thread.c:301
      #4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 16 (Thread 30616):
      #0 0x000000382869a541 in nanosleep () from /lib64/libc.so.6
      #1 0x00000038286cded4 in usleep () from /lib64/libc.so.6
      #2 0x00002aaaaaf2f255 in updateStatsThread (arg=0x42e84c0) at src/memory_tracker.cc:31
      #3 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #4 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 15 (Thread 30617):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b98090, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b98090, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b96c40) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b96c40) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 14 (Thread 30618):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b982d0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b982d0, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b96a80) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b96a80) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 13 (Thread 30619):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b98240, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b98240, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b97880) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b97880) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 12 (Thread 30620):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b981b0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b981b0, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b976c0) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b976c0) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 11 (Thread 30621):

      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf0fd7f in wait (this=0x8ba4000) at src/syncobject.hh:57
      #2 wait (this=0x8ba4000) at src/syncobject.hh:73
      --Type <return> to continue, or q <return> to quit--
      #3 wait (this=0x8ba4000) at src/tapconnmap.hh:169
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x8ba4000) at src/ep_engine.cc:3369
      #5 0x00002aaaaaf0fe63 in EvpNotifyPendingConns (arg=0x8ba4000) at src/ep_engine.cc:1121
      #6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0

      #7 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 10 (Thread 30630):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b985a0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b985a0, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfa700) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfa700) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 9 (Thread 30631):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b986c0, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b986c0, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfa540) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfa540) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 8 (Thread 30632):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b98630, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b98630, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfb340) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfb340) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 7 (Thread 30633):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b98870, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b98870, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfb180) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfb180) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 6 (Thread 30634):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf0fd7f in wait (this=0x8ba5b00) at src/syncobject.hh:57
      #2 wait (this=0x8ba5b00) at src/syncobject.hh:73
      #3 wait (this=0x8ba5b00) at src/tapconnmap.hh:169
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x8ba5b00) at src/ep_engine.cc:3369
      #5 0x00002aaaaaf0fe63 in EvpNotifyPendingConns (arg=0x8ba5b00) at src/ep_engine.cc:1121
      #6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #7 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 5 (Thread 30681):
      #0 0x000000382920e498 in pread64 () from /lib64/libpthread.so.0
      #1 0x00002aaaab1cb738 in couch_pread (handle=<value optimized out>, buf=0x39510c31, nbyte=8192, offset=1982464) at src/os.c:38
      --Type <return> to continue, or q <return> to quit--
      #2 0x00002aaaaaf6e13d in cfs_pread (h=0x40456020, buf=0x39510c31, sz=8192, off=1982464) at src/couch-kvstore/couch-fs-stats.cc:70
      #3 0x00002aaaab1c8a2b in load_buffer_from (handle=0x2aaac0f09bc0, buf=<value optimized out>, nbyte=2, offset=<value optimized out>) at src/iobuffer.c:153
      #4 buffered_pread (handle=0x2aaac0f09bc0, buf=<value optimized out>, nbyte=2, offset=<value optimized out>) at src/iobuffer.c:292
      #5 0x00002aaaab1c5323 in find_header_at_pos (filename=<value optimized out>, flags=<value optimized out>, ops=<value optimized out>, pDb=0x4ceced90) at src/couch_db.c:42
      #6 find_header (filename=<value optimized out>, flags=<value optimized out>, ops=<value optimized out>, pDb=0x4ceced90) at src/couch_db.c:88
      #7 couchstore_open_db_ex (filename=<value optimized out>, flags=<value optimized out>, ops=<value optimized out>, pDb=0x4ceced90) at src/couch_db.c:250
      #8 0x00002aaaaaf64974 in CouchKVStore::openDB (this=0x8bb9b00, vbucketId=374, fileRev=2, db=0x4ceced90, options=1, newFileRev=0x4ceced88) at src/couch-kvstore/couch-kvstore.cc:1129
      #9 0x00002aaaaaf67c4e in CouchKVStore::setVBucketState (this=0x8bb9b00, vbucketId=374, vbstate=..., vb_change_type=1, newfile=128, notify=true) at src/couch-kvstore/couch-kvstore.cc:706
      #10 0x00002aaaaaf68a39 in CouchKVStore::snapshotVBuckets (this=0x8bb9b00, vbstates=std::map with 833 elements =

      {...}

      ) at src/couch-kvstore/couch-kvstore.cc:596
      #11 0x00002aaaaaefb0fe in EventuallyPersistentStore::snapshotVBuckets (this=0x8ba3600, priority=...) at src/ep.cc:845
      #12 0x00002aaaaaf09d41 in SnapshotVBucketsCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /opt/couchbase/lib/memcached/ep.so
      #13 0x00002aaaaaef522a in Dispatcher::run (this=0x9368000) at src/dispatcher.cc:171
      #14 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368000) at src/dispatcher.cc:28
      #15 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #16 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 4 (Thread 30682):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaef29f8 in wait (this=0x8b98c60, d=...) at src/syncobject.hh:57
      #2 IdleTask::run (this=0x8b98c60, d=...) at src/dispatcher.cc:328
      #3 0x00002aaaaaef522a in Dispatcher::run (this=0x9368e00) at src/dispatcher.cc:171
      #4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368e00) at src/dispatcher.cc:28
      #5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 3 (Thread 30683):
      #0 0x000000382920e498 in pread64 () from /lib64/libpthread.so.0
      #1 0x00002aaaab1cb738 in couch_pread (handle=<value optimized out>, buf=0x2aaad9b7c031, nbyte=8192, offset=1376256) at src/os.c:38
      #2 0x00002aaaaaf6e13d in cfs_pread (h=0x9d75680, buf=0x2aaad9b7c031, sz=8192, off=1376256) at src/couch-kvstore/couch-fs-stats.cc:70
      #3 0x00002aaaab1c8a2b in load_buffer_from (handle=0xcf07d70, buf=<value optimized out>, nbyte=1996, offset=<value optimized out>) at src/iobuffer.c:153
      #4 buffered_pread (handle=0xcf07d70, buf=<value optimized out>, nbyte=1996, offset=<value optimized out>) at src/iobuffer.c:292
      #5 0x00002aaaab1c7512 in read_skipping_prefixes (file=0x2aaac5ac12d0, pos=1376257, ret_ptr=0x4e2d02e8, header=<value optimized out>) at src/couch_file_read.c:68
      #6 pread_bin_internal (file=0x2aaac5ac12d0, pos=1376257, ret_ptr=0x4e2d02e8, header=<value optimized out>) at src/couch_file_read.c:111
      #7 0x00002aaaab1c7627 in pread_compressed (file=<value optimized out>, pos=<value optimized out>, ret_ptr=0x4e2d0328) at src/couch_file_read.c:133
      #8 0x00002aaaab1c495a in bp_to_doc (db=<value optimized out>, docinfo=0x2aaae1ae4090, pDoc=0x4e2d03d8, options=<value optimized out>) at src/couch_db.c:403
      #9 couchstore_open_doc_with_docinfo (db=<value optimized out>, docinfo=0x2aaae1ae4090, pDoc=0x4e2d03d8, options=<value optimized out>) at src/couch_db.c:549
      #10 0x00002aaaaaf611a5 in CouchKVStore::recordDbDump (db=0x2aaac5ac12d0, docinfo=0x2aaae1ae4090, ctx=0x4e2d0d10) at src/couch-kvstore/couch-kvstore.cc:1342
      #11 0x00002aaaab1c62bd in lookup_callback (rq=<value optimized out>, k=0x4e2d0510, v=0x4e2d0500) at src/couch_db.c:638
      #12 0x00002aaaab1c3064 in btree_lookup_inner (rq=0x4e2d0750, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:78
      #13 0x00002aaaab1c2f58 in btree_lookup_inner (rq=0x4e2d0750, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52
      #14 0x00002aaaab1c2f58 in btree_lookup_inner (rq=0x4e2d0750, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52
      #15 0x00002aaaab1c478c in couchstore_changes_since (db=<value optimized out>, since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>, ctx=<value optimized out>)
      at src/couch_db.c:682
      #16 0x00002aaaaaf65e2c in CouchKVStore::loadDB (this=0x9cf0000, cb=std::tr1::shared_ptr (count 5) 0xb697860, keysOnly=false, vbids=<value optimized out>, options=0)
      at src/couch-kvstore/couch-kvstore.cc:962
      #17 0x00002aaaaaf66913 in CouchKVStore::dump (this=0x9cf0000, vb=414, cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:780
      #18 0x00002aaaaaee458a in BackfillDiskLoad::callback (this=0x2aaac5f12b40, d=<value optimized out>, t=<value optimized out>) at src/backfill.cc:60
      #19 0x00002aaaaaef522a in Dispatcher::run (this=0x9368c40) at src/dispatcher.cc:171
      #20 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368c40) at src/dispatcher.cc:28
      #21 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #22 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 2 (Thread 30685):
      #0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002aaaaaf0fd7f in wait (this=0x8ba7600) at src/syncobject.hh:57
      --Type <return> to continue, or q <return> to quit--
      #2 wait (this=0x8ba7600) at src/syncobject.hh:73
      #3 wait (this=0x8ba7600) at src/tapconnmap.hh:169
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x8ba7600) at src/ep_engine.cc:3369
      #5 0x00002aaaaaf0fe63 in EvpNotifyPendingConns (arg=0x8ba7600) at src/ep_engine.cc:1121
      #6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #7 0x00000038286d44bd in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x4ecd2940 (LWP 30684)):
      #0 0x00002aaaaaf32d11 in HashTable::visit (this=0x2aaac5cce408, visitor=...) at src/stored-value.cc:395
      #1 0x00002aaaaaef8e62 in VBCBAdaptor::callback (this=0x3907f7a0, d=..., t=...) at src/ep.cc:2522
      #2 0x00002aaaaaef522a in Dispatcher::run (this=0x9368a80) at src/dispatcher.cc:171
      #3 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368a80) at src/dispatcher.cc:28
      #4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000038286d44bd in clone () from /lib64/libc.so.6

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        andreibaranouski Andrei Baranouski added a comment -

        the solution is upgrade paramiko on 1.9.0

        cd /tmp;rm -rf paramiko*;
        wget http://pypi.python.org/packages/source/p/paramiko/paramiko-1.9.0.tar.gz;
        tar -xvf paramiko-1.9.0.tar.gz;
        cd paramiko-1.9.0;
        python setup.py build;python setup.py install;

        then you can check version as:

        root@slv-1901:~# python
        Python 2.6.5 (r265:79063, Oct 1 2012, 22:04:36)
        [GCC 4.4.3] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> import paramiko
        >>> paramiko._version_
        '1.9.0'

        Chiyoung, let me know if there's anything else I can do

        Show
        andreibaranouski Andrei Baranouski added a comment - the solution is upgrade paramiko on 1.9.0 cd /tmp;rm -rf paramiko*; wget http://pypi.python.org/packages/source/p/paramiko/paramiko-1.9.0.tar.gz ; tar -xvf paramiko-1.9.0.tar.gz; cd paramiko-1.9.0; python setup.py build;python setup.py install; then you can check version as: root@slv-1901:~# python Python 2.6.5 (r265:79063, Oct 1 2012, 22:04:36) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import paramiko >>> paramiko._ version _ '1.9.0' Chiyoung, let me know if there's anything else I can do
        Hide
        chiyoung Chiyoung Seo added a comment -

        Thanks Andrei for your help.

        I'm running the same test repeatedly now, but didn't see any seg fault crash so far. I will keep running it continuously and update the ticket later.

        Show
        chiyoung Chiyoung Seo added a comment - Thanks Andrei for your help. I'm running the same test repeatedly now, but didn't see any seg fault crash so far. I will keep running it continuously and update the ticket later.
        Hide
        mikew Mike Wiederhold added a comment -

        Chiyoung,

        Have you been able to reproduce this? If not, I think it might makes sense to close this as cannot reproduce and see if it comes up again. Also note I fixed that memory corruption issue a few days ago and that could have possibly been related to this issue. Anyways, let me know what you think.

        Show
        mikew Mike Wiederhold added a comment - Chiyoung, Have you been able to reproduce this? If not, I think it might makes sense to close this as cannot reproduce and see if it comes up again. Also note I fixed that memory corruption issue a few days ago and that could have possibly been related to this issue. Anyways, let me know what you think.
        Hide
        chiyoung Chiyoung Seo added a comment -

        Mike,

        I don't think your fix resolved the memory corruption one. Tap Producer always checkes the existence of a given vbucket instance before getting the item from it.

        I ran the same test for three days, but didn't see the crash. I closed this issue at this time and marked it as "can't reproduce".

        Show
        chiyoung Chiyoung Seo added a comment - Mike, I don't think your fix resolved the memory corruption one. Tap Producer always checkes the existence of a given vbucket instance before getting the item from it. I ran the same test for three days, but didn't see the crash. I closed this issue at this time and marked it as "can't reproduce".
        Hide
        maria Maria McDuff (Inactive) added a comment -

        NOT REPRODUCIBLE.
        CLOSING.

        Show
        maria Maria McDuff (Inactive) added a comment - NOT REPRODUCIBLE. CLOSING.

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes