Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11285

memcached crashed (segfault) after online upgrade from 2.0.0 to 3.0.0-764 failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Test Blocker
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • centos 5.8

    Description

      Install couchbase server 2.0.0 on 2 nodes
      1:10.6.2.112
      2:10.6.2.113

      Create cluster 2.0.0 on those 2 nodes
      Create a default bucket
      Create 2 docs with 2 views per doc

      Install couchbase server 3.0.0-764 on 2 nodes
      3:10.3.3.224
      4:10.3.121.241

      Add node 224 and 241 into cluster 2.0.0
      Rebalance in 2 nodes 3.0.0-764 into cluster 2.0.0 passed.

      Remove 2 2.0.0 nodes out of cluster.
      Rebalance failed as saying in bug MB-11176
      Moment later, I saw memcached crashed on nodes 10.3.121.241 (with cb version 3.0.0-764)

      warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff0cffd000
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
      Program terminated with signal 11, Segmentation fault.
      #0 0x00002aaaaaee778b in CheckpointManager::addStats (this=0x25290018, add_stat=0x407680 <append_stats>, cookie=0x24e5aa00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc:1573
      1573 /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc: No such file or directory.
      in /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc

      Thread 13 (Thread 0x2b9fdead6750 (LWP 18909)):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6000, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6000, flags=<value optimized out>) at event.c:1558
      #3 0x000000000040f169 in main (argc=<value optimized out>, argv=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8786

      Thread 12 (Thread 18911):
      #0 0x0000003866cc545b in read () from /lib64/libc.so.6
      #1 0x0000003866c6b677 in _IO_new_file_underflow () from /lib64/libc.so.6
      #2 0x0000003866c6c03e in _IO_default_uflow_internal () from /lib64/libc.so.6
      #3 0x0000003866c61124 in _IO_getline_info_internal () from /lib64/libc.so.6
      #4 0x0000003866c5ffc9 in fgets () from /lib64/libc.so.6
      #5 0x00002b9fdead78b1 in check_stdin_thread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
      #6 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a0e0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #7 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #8 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 11 (Thread 18913):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x2aaaaacbee60, mutex=0x2aaaaacbee20, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
      #2 0x00002aaaaaaae548 in logger_thead_main (arg=0x20612ac0) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a080) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 10 (Thread 18930):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6280, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6280, flags=<value optimized out>) at event.c:1558
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a100) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 9 (Thread 18931):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6500, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6500, flags=<value optimized out>) at event.c:1558
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a110) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 8 (Thread 18932):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6780, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6780, flags=<value optimized out>) at event.c:1558
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a120) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 7 (Thread 18933):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6a00, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6a00, flags=<value optimized out>) at event.c:1558
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a130) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 6 (Thread 18934):
      #0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6c80, tv=<value optimized out>) at epoll.c:404
      #2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6c80, flags=<value optimized out>) at event.c:1558
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a140) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 5 (Thread 19231):
      #0 0x0000003866c99221 in nanosleep () from /lib64/libc.so.6
      #1 0x0000003866cccba4 in usleep () from /lib64/libc.so.6
      #2 0x00002aaaaaf45155 in updateStatsThread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
      #3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a1f0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 4 (Thread 19243):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x24f1d940, mutex=0x24f1d908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
      #2 0x00002aaaaaf36577 in wait (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
      #3 ExecutorPool::trySleep (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
      #4 0x00002aaaaaf367c5 in ExecutorPool::nextTask (this=0x24f1d8e0, t=..., tick=24 '\030') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
      #5 0x00002aaaaaf475ad in ExecutorThread::run (this=0x2066a960) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
      #6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x24f1d944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
      #7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060abc0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 3 (Thread 19244):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x24f1d940, mutex=0x24f1d908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
      #2 0x00002aaaaaf36577 in wait (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
      #3 ExecutorPool::trySleep (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
      #4 0x00002aaaaaf367aa in ExecutorPool::nextTask (this=0x24f1d8e0, t=..., tick=120 'x') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:135
      #5 0x00002aaaaaf475ad in ExecutorThread::run (this=0x2066a9b0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
      #6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x24f1d944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
      #7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060abb0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 2 (Thread 19245):
      #0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x24f1d940, mutex=0x24f1d908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
      #2 0x00002aaaaaf36577 in wait (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
      #3 ExecutorPool::trySleep (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
      #4 0x00002aaaaaf367c5 in ExecutorPool::nextTask (this=0x24f1d8e0, t=..., tick=16 '\020') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
      #5 0x00002aaaaaf475ad in ExecutorThread::run (this=0x2066aa00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
      #6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x24f1d944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
      #7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060abd0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003866cd325d in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x462ac940 (LWP 19242)):
      #0 0x00002aaaaaee778b in CheckpointManager::addStats (this=0x25290018, add_stat=0x407680 <append_stats>, cookie=0x24e5aa00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc:1573
      #1 0x00002aaaaaf32ac8 in addCheckpointStat (this=<value optimized out>, vb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3508
      #2 StatCheckpointVisitor::visitBucket (this=<value optimized out>, vb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3492
      #3 0x00002aaaaaef5177 in EventuallyPersistentStore::visit (this=<value optimized out>, visitor=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:2768
      #4 0x00002aaaaaf34006 in StatCheckpointTask::run (this=0x24ef57a0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3528
      #5 0x00002aaaaaf47641 in ExecutorThread::run (this=0x2066a910) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
      #6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x462abc00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
      #7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060aba0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
      #8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003866cd325d in clone () from /lib64/libc.so.6
      --------------------------------------------------------------------------------
      Module information:
      /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0:
      -rwxr-xr-x 1 bin bin 407937 Jun 2 07:48 /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
      d821e25dc3096566390192604386236c /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
      /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1:

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sundar Sundar Sridharan (Inactive)
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty