Details
-
Bug
-
Resolution: Duplicate
-
Major
-
Cheshire-Cat
-
7.0.0-2017
-
Untriaged
-
-
1
-
Unknown
Description
Steps:
1. Create a 10 node cluster.
2. Create a default bucket with replica=0
3. Load data
4. While data loading is happening kill -9 memcached on all the nodes.
5. Core dumps observed are showing crashes.
AssertionError: Test failed, Coredump found on servers ['172.23.120.63', '172.23.120.66']
memcached logs:
2020-05-13T04:17:46.036608-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2017). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/8cd50637-26c9-4317-c0cece83-56c1299f.dmp before terminating.
|
2020-05-13T04:17:46.036633-07:00 CRITICAL Stack backtrace of crashed thread:
|
2020-05-13T04:17:46.037063-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x13946d]
|
2020-05-13T04:17:46.037108-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x14f1ba]
|
2020-05-13T04:17:46.037117-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x14f4f8]
|
2020-05-13T04:17:46.037124-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fcfe86be000+0xf5f0]
|
2020-05-13T04:17:46.037180-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x882ed]
|
2020-05-13T04:17:46.037195-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x7e37c]
|
2020-05-13T04:17:46.037203-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x80b87]
|
2020-05-13T04:17:46.037209-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x84328]
|
2020-05-13T04:17:46.037214-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x84a65]
|
2020-05-13T04:17:46.037220-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7fcfea141000+0xed8e]
|
2020-05-13T04:17:46.037225-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7fcfea141000+0x17d01]
|
2020-05-13T04:17:46.037231-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7fcfea141000+0x1854f]
|
2020-05-13T04:17:46.037238-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x69179]
|
2020-05-13T04:17:46.037246-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fcfeb078000+0x10827]
|
2020-05-13T04:17:46.037253-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fcfe86be000+0x7e65]
|
172.23.120.63 : Stack Trace of 8cd50637-26c9-4317-c0cece83-56c1299f.dmp
/opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/8cd50637-26c9-4317-c0cece83-56c1299f.dmp > /opt/couchbase/var/lib/couchbase/crash/8cd50637-26c9-4317-c0cece83-56c1299f.core
|
gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/8cd50637-26c9-4317-c0cece83-56c1299f.core -ex bt all -ex quit
|
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach’.
|
#0 0x00000000004882ed in std::deque<std::unique_ptr<Cookie, std::default_delete<Cookie> >, std::allocator<std::unique_ptr<Cookie, std::default_delete<Cookie> > > >::_M_erase(std::_Deque_iterator<std::unique_ptr<Cookie, std::default_delete<Cookie> >, std::unique_ptr<Cookie, std::default_delete<Cookie> >&, std::unique_ptr<Cookie, std::default_delete<Cookie> >*>) ()
|
#0 0x00000000004882ed in std::deque<std::unique_ptr<Cookie, std::default_delete<Cookie> >, std::allocator<std::unique_ptr<Cookie, std::default_delete<Cookie> > > >::_M_erase(std::_Deque_iterator<std::unique_ptr<Cookie, std::default_delete<Cookie> >, std::unique_ptr<Cookie, std::default_delete<Cookie> >&, std::unique_ptr<Cookie, std::default_delete<Cookie> >*>) ()
|
#1 0x000000000047e37c in Connection::processAllReadyCookies() ()
|
#2 0x0000000000480b87 in Connection::executeCommandPipeline() ()
|
#3 0x0000000000484328 in Connection::executeCommandsCallback() ()
|
#4 0x0000000000484a65 in Connection::rw_callback(bufferevent*, void*) ()
|
#5 0x00007fcfea14fd8e in bufferevent_run_deferred_callbacks_unlocked () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
|
#6 0x00007fcfea158d01 in event_process_active_single_queue.isra.31 () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
|
#7 0x00007fcfea15954f in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
|
#8 0x0000000000469179 in worker_libevent(void*) ()
|
#9 0x00007fcfeb088827 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
|
#10 0x00007fcfe86c5e65 in start_thread () from /lib64/libpthread.so.0
|
#11 0x00007fcfe83ee88d in clone () from /lib64/libc.so.6
|
Note: The bucket is recovered after the crash but we end the test as in we see core dumps.
Attachments
Issue Links
- duplicates
-
-
- Closed
-