Details
-
Bug
-
Resolution: Cannot Reproduce
-
Test Blocker
-
3.0
-
Security Level: Public
-
None
-
centos 5.8
-
Untriaged
-
Centos 64-bit
-
-
Yes
-
June 30 - July 18
Description
Install couchbase server 2.0.0 on 2 nodes
1:10.6.2.112
2:10.6.2.113
Create cluster 2.0.0 on those 2 nodes
Create a default bucket
Create 2 docs with 2 views per doc
Install couchbase server 3.0.0-764 on 2 nodes
3:10.3.3.224
4:10.3.121.241
Add node 224 and 241 into cluster 2.0.0
Rebalance in 2 nodes 3.0.0-764 into cluster 2.0.0 passed.
Remove 2 2.0.0 nodes out of cluster.
Rebalance failed as saying in bug MB-11176
Moment later, I saw memcached crashed on nodes 10.3.121.241 (with cb version 3.0.0-764)
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff0cffd000
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 0x00002aaaaaee778b in CheckpointManager::addStats (this=0x25290018, add_stat=0x407680 <append_stats>, cookie=0x24e5aa00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc:1573
1573 /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc: No such file or directory.
in /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc
Thread 13 (Thread 0x2b9fdead6750 (LWP 18909)):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6000, tv=<value optimized out>) at epoll.c:404
#2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040f169 in main (argc=<value optimized out>, argv=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8786
Thread 12 (Thread 18911):
#0 0x0000003866cc545b in read () from /lib64/libc.so.6
#1 0x0000003866c6b677 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003866c6c03e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003866c61124 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x0000003866c5ffc9 in fgets () from /lib64/libc.so.6
#5 0x00002b9fdead78b1 in check_stdin_thread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a0e0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#7 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 11 (Thread 18913):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x2aaaaacbee60, mutex=0x2aaaaacbee20, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00002aaaaaaae548 in logger_thead_main (arg=0x20612ac0) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a080) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 10 (Thread 18930):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6280, tv=<value optimized out>) at epoll.c:404
#2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6280, flags=<value optimized out>) at event.c:1558
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a100) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 9 (Thread 18931):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6500, tv=<value optimized out>) at epoll.c:404
#2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6500, flags=<value optimized out>) at event.c:1558
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a110) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 8 (Thread 18932):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6780, tv=<value optimized out>) at epoll.c:404
#2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6780, flags=<value optimized out>) at event.c:1558
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a120) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 7 (Thread 18933):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6a00, flags=<value optimized out>) at event.c:1558
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a130) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 6 (Thread 18934):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b9fde8b9376 in epoll_dispatch (base=0x24ec6c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002b9fde8a4c44 in event_base_loop (base=0x24ec6c80, flags=<value optimized out>) at event.c:1558
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a140) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 5 (Thread 19231):
#0 0x0000003866c99221 in nanosleep () from /lib64/libc.so.6
#1 0x0000003866cccba4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf45155 in updateStatsThread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060a1f0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 4 (Thread 19243):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x24f1d940, mutex=0x24f1d908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00002aaaaaf36577 in wait (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00002aaaaaf367c5 in ExecutorPool::nextTask (this=0x24f1d8e0, t=..., tick=24 '\030') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00002aaaaaf475ad in ExecutorThread::run (this=0x2066a960) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x24f1d944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060abc0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 3 (Thread 19244):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x24f1d940, mutex=0x24f1d908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00002aaaaaf36577 in wait (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00002aaaaaf367aa in ExecutorPool::nextTask (this=0x24f1d8e0, t=..., tick=120 'x') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:135
#5 0x00002aaaaaf475ad in ExecutorThread::run (this=0x2066a9b0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x24f1d944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060abb0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 2 (Thread 19245):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002b9fdde358eb in cb_cond_timedwait (cond=0x24f1d940, mutex=0x24f1d908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00002aaaaaf36577 in wait (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x24f1d8e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00002aaaaaf367c5 in ExecutorPool::nextTask (this=0x24f1d8e0, t=..., tick=16 '\020') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00002aaaaaf475ad in ExecutorThread::run (this=0x2066aa00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x24f1d944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060abd0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003866cd325d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x462ac940 (LWP 19242)):
#0 0x00002aaaaaee778b in CheckpointManager::addStats (this=0x25290018, add_stat=0x407680 <append_stats>, cookie=0x24e5aa00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/checkpoint.cc:1573
#1 0x00002aaaaaf32ac8 in addCheckpointStat (this=<value optimized out>, vb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3508
#2 StatCheckpointVisitor::visitBucket (this=<value optimized out>, vb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3492
#3 0x00002aaaaaef5177 in EventuallyPersistentStore::visit (this=<value optimized out>, visitor=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:2768
#4 0x00002aaaaaf34006 in StatCheckpointTask::run (this=0x24ef57a0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3528
#5 0x00002aaaaaf47641 in ExecutorThread::run (this=0x2066a910) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#6 0x00002aaaaaf47b56 in launch_executor_thread (arg=0x462abc00) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00002b9fdde35b6f in platform_thread_wrap (arg=0x2060aba0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003866cd325d in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0:
-rwxr-xr-x 1 bin bin 407937 Jun 2 07:48 /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
d821e25dc3096566390192604386236c /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
/opt/couchbase/bin/../lib/libcbsasl.so.1.1.1:
Attachments
Issue Links
- blocks
-
MB-11176 online upgrade from 2.0.0 to 3.0.0-713 failed due to view engine crashed
- Closed