Details
-
Bug
-
Resolution: User Error
-
Major
-
Cheshire-Cat
-
Couchbase version 7.0.0-2007
Ubuntu 18.04.4 LTS
-
Untriaged
-
1
-
Unknown
Description
Crash found during kv-engine-cheshire-cat-jepsen-nightly for build 7.0.0-2007 on node 172.23.106.61.
During test:
lein trampoline run test --nodes-file ./nodes --username root --password couchbase --workload=rebalance --scenario=sequential-rebalance-out-in --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-6.log
CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2007). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/19af5f74-43e9-45f7-3f3f60b4-50d01e74.dmp before terminating.
|
CRITICAL Stack backtrace of crashed thread:
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1394fd]
|
CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x14f24a]
|
CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x14f588]
|
CRITICAL /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f2aee9bf000+0x12890]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x14d4b7]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x6d4a9]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x96d88]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x9812e]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x80cbb]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x843b8]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x84af5]
|
CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f2af04dc000+0xed1e]
|
CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f2af04dc000+0x17c91]
|
CRITICAL /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7f2af04dc000+0x184df]
|
CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x69299]
|
CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f2af1411000+0x10937]
|
CRITICAL /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f2aee9bf000+0x76db]
|
CRITICAL /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f2aee5ce000+0x12188f]
|
(gdb) bt
|
#0 get_bucket_index (h=<optimized out>, h=<optimized out>, h=<optimized out>, value=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/HdrHistogram_c/src/hdr_histogram.c:148
|
#1 counts_index_for (value=417, h=0x0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/HdrHistogram_c/src/hdr_histogram.c:176
|
#2 hdr_record_values (h=0x0, value=417, count=1) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/HdrHistogram_c/src/hdr_histogram.c:489
|
#3 0x000000000046d4a9 in Hdr1sfMicroSecHistogram::add (count=1, v=..., this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/hdrhistogram.h:377
|
#4 Timings::collect (this=0x7fa9e8 <all_buckets+232>, opcode=opcode@entry=cb::mcbp::ClientOpcode::Set, nsec=nsec@entry=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/timings.cc:52
|
#5 0x0000000000496d88 in Cookie::collectTimings (this=this@entry=0x7f2aeae71c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:557
|
#6 0x000000000049812e in Cookie::execute (this=this@entry=0x7f2aeae71c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:160
|
#7 0x0000000000480cbb in Connection::executeCommandPipeline (this=this@entry=0x7f2aeacc5f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:569
|
#8 0x00000000004843b8 in Connection::executeCommandsCallback (this=this@entry=0x7f2aeacc5f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:638
|
#9 0x0000000000484af5 in Connection::rw_callback (ctx=0x7f2aeacc5f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:762
|
#10 0x00007f2af04ead1e in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7f2ad8003bf0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208
|
#11 0x00007f2af04f3c91 in event_process_active_single_queue (base=base@entry=0x15049b0, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726
|
#12 0x00007f2af04f44df in event_process_active (base=0x15049b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789
|
#13 event_base_loop (base=0x15049b0, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012
|
#14 0x0000000000469299 in worker_libevent (arg=0x7f2aedff3550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:210
|
#15 0x00007f2af1421937 in CouchbaseThread::run (this=0x7f2aeaca7850) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
|
#16 platform_thread_wrap (arg=0x7f2aeaca7850) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
|
#17 0x00007f2aee9c66db in start_thread (arg=0x7f2ae9477700) at pthread_create.c:463
|
#18 0x00007f2aee6ef88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
Zip of logs, pcap and dump file can be found here https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-39284/1cf50987/20200507T045459.000-0700.zip