Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Couchbase EE 7.0.0-4476
-
Triaged
-
Centos 64-bit
-
-
1
-
Yes
-
KV-Engine 2021-Feb
Description
Script to Repo
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.23984.ini GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE_dgm,rerun=False,get-cbcollect-info=True,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-4476 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=dgm.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,dgm=60,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE_dgm'
|
The test does:
1. creates 5 init nodes
2. creates 1 CB bucket with collections
3. Loads the bucket into a dgm of 60%
4. Rebalances out 2 nodes with parallel doc ops (at the above mentioned durability level) and creating and dropping collections
Rebalance seems to have failed and a minidump is observed
(unfortunately don't have the timeline as it was run at error log level)
2021-02-17 17:31:03,336 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:718] 172.23.105.192: 1 core dump seen
|
on .192 memcached log
2021-02-17T17:30:16.970396-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-4476). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c6553ed9-37b3-42e0-ce7e1dbb-6dcc5cda.dmp before terminating. |
2021-02-17T17:30:16.970469-08:00 CRITICAL Stack backtrace of crashed thread: |
2021-02-17T17:30:16.971010-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x14c8fd] |
2021-02-17T17:30:16.971055-08:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x162cea] |
2021-02-17T17:30:16.971075-08:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x163028] |
2021-02-17T17:30:16.971100-08:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f6509ff7000+0xf5e0] |
2021-02-17T17:30:16.971124-08:00 CRITICAL #4 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x107993] |
2021-02-17T17:30:16.971152-08:00 CRITICAL #5 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x106f80] |
2021-02-17T17:30:16.971180-08:00 CRITICAL #6 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x1001e0] |
2021-02-17T17:30:16.971204-08:00 CRITICAL #7 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0xfafb7] |
2021-02-17T17:30:16.971328-08:00 CRITICAL #8 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x104aa0] |
2021-02-17T17:30:16.971352-08:00 CRITICAL #9 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x10aca9] |
2021-02-17T17:30:16.971372-08:00 CRITICAL #10 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x1d78df] |
2021-02-17T17:30:16.971397-08:00 CRITICAL #11 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x1d9fc3] |
2021-02-17T17:30:16.971418-08:00 CRITICAL #12 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x1e7674] |
2021-02-17T17:30:16.971453-08:00 CRITICAL #13 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x197d8e] |
2021-02-17T17:30:16.971486-08:00 CRITICAL #14 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x198444] |
2021-02-17T17:30:16.971513-08:00 CRITICAL #15 /opt/couchbase/bin/../lib/libep.so() [0x7f650de59000+0x129bc4] |
2021-02-17T17:30:16.971634-08:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x50b61] |
2021-02-17T17:30:16.971664-08:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0xd6d07] |
2021-02-17T17:30:16.971687-08:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0xd6ac8] |
2021-02-17T17:30:16.971702-08:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0xdc4a0] |
2021-02-17T17:30:16.971725-08:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0xb0766] |
2021-02-17T17:30:16.971755-08:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0xb4436] |
2021-02-17T17:30:16.971776-08:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0xa24d9] |
2021-02-17T17:30:16.971805-08:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x838ad] |
2021-02-17T17:30:16.971817-08:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x8a8d5] |
2021-02-17T17:30:16.971837-08:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x8b78f] |
2021-02-17T17:30:16.971852-08:00 CRITICAL #26 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f650b878000+0xed8e] |
2021-02-17T17:30:16.971872-08:00 CRITICAL #27 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f650b878000+0x17d01] |
2021-02-17T17:30:16.971882-08:00 CRITICAL #28 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7f650b878000+0x1854f] |
2021-02-17T17:30:16.971965-08:00 CRITICAL #29 /opt/couchbase/bin/memcached() [0x400000+0x6bba9] |
2021-02-17T17:30:16.971992-08:00 CRITICAL #30 /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f650c7b0000+0x16727] |
2021-02-17T17:30:16.972014-08:00 CRITICAL #31 /lib64/libpthread.so.0() [0x7f6509ff7000+0x7e25] |
2021-02-17T17:30:16.972101-08:00 CRITICAL #32 /lib64/libc.so.6(clone+0x6d) [0x7f6509c34000+0xf834d] |
Logs attached.
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
| For Gerrit Dashboard: MB-44414 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 146563,3 | MB-44414: Actually invalidate chains when moving writes to resolvedQ | master | kv_engine | Status: MERGED | +2 | +1 |