Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
Cheshire-Cat
-
7.0.0-4532
-
Triaged
-
Centos 64-bit
-
1
-
Yes
Description
Script to Repro
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.14632.ini GROUP=durability_majority_dgm,rerun=False,upgrade_version=7.0.0-4532 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=dgm.buckets_for_rebalance_tests,data_load_stage=during,dgm=60,skip_validations=False,GROUP=durability_majority_dgm'
|
Steps to Repro
1) Create a 4 node cluster
----------------------------------------------------------------------
Nodes | Services | Version | CPU | Status |
----------------------------------------------------------------------
172.23.121.250 | kv | 7.0.0-4532-enterprise | 1.42821348033 | Cluster node |
172.23.121.253 | None | <--- IN — | ||
172.23.121.247 | None | <--- IN — | ||
172.23.121.248 | None | <--- IN — |
----------------------------------------------------------------------
2) Create bucket/scope/collections/data
2021-02-24 01:39:16,548 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-------------------------------------------------------------------------
Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-------------------------------------------------------------------------
default | couchbase | 2 | none | 0 | 600000 | 1258291200 | 614504792 | 565364241 |
-------------------------------------------------------------------------
3) Push the bucket to DGM
2021-02-24 01:40:32,737 | test | INFO | pool-5-thread-23 | [task:_load_bucket_into_dgm:2079] Active DGM 100% Replica DGM 51.8018674882% achieved for 'default'. Loaded docs: 1240000
|
4) Add 172.23.121.251 and 172.23.121.252, remove 172.23.121.248 and do rebalance
2021-02-24 01:40:41,663 | test | INFO | pool-5-thread-8 | [table_view:display:72] Rebalance Overview
-----------------------------------------------------------------------
Nodes | Services | Version | CPU | Status |
-----------------------------------------------------------------------
172.23.121.251 | kv | 7.0.0-4532-enterprise | 0 | Cluster node |
172.23.121.247 | kv | 7.0.0-4532-enterprise | 66.4871794872 | Cluster node |
172.23.121.253 | kv | 7.0.0-4532-enterprise | 66.9830682401 | Cluster node |
172.23.121.250 | kv | 7.0.0-4532-enterprise | 76.4751154438 | Cluster node |
172.23.121.248 | kv | 7.0.0-4532-enterprise | 66.6238767651 | — OUT ---> |
172.23.121.252 | kv | 7.0.0-4532-enterprise | 0.552902739382 | Cluster node |
-----------------------------------------------------------------------
Rebalance fails with minidumps seen on 172.23.121.247(182b5464-e6d1-4476-ded9b1b6-eedc363a.dmp)
grep CRITICAL on 172.23.121.247
Balakumarans-MacBook-Pro-2:cbcollect_info_ns_1@172.23.121.247_20210224-094114 balakumaran.g$ grep CRITICAL memcached*
|
memcached.log:2021-02-24T01:40:50.684860-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-4532). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/182b5464-e6d1-4476-ded9b1b6-eedc363a.dmp before terminating.
|
memcached.log:2021-02-24T01:40:50.684884-08:00 CRITICAL Stack backtrace of crashed thread:
|
memcached.log:2021-02-24T01:40:50.685162-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x14ca3d]
|
memcached.log:2021-02-24T01:40:50.685180-08:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x162e3a]
|
memcached.log:2021-02-24T01:40:50.685190-08:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x163178]
|
memcached.log:2021-02-24T01:40:50.685197-08:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fca9315b000+0xf5f0]
|
memcached.log:2021-02-24T01:40:50.685223-08:00 CRITICAL #4 /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt11_Hash_bytesPKvmm+0x30) [0x7fca93890000+0x8f570]
|
memcached.log:2021-02-24T01:40:50.685234-08:00 CRITICAL #5 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x10783f]
|
memcached.log:2021-02-24T01:40:50.685242-08:00 CRITICAL #6 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x106e20]
|
memcached.log:2021-02-24T01:40:50.685251-08:00 CRITICAL #7 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x100080]
|
memcached.log:2021-02-24T01:40:50.685258-08:00 CRITICAL #8 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0xfae77]
|
memcached.log:2021-02-24T01:40:50.685264-08:00 CRITICAL #9 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x104940]
|
memcached.log:2021-02-24T01:40:50.685270-08:00 CRITICAL #10 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x10ab6c]
|
memcached.log:2021-02-24T01:40:50.685281-08:00 CRITICAL #11 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x1d74df]
|
memcached.log:2021-02-24T01:40:50.685289-08:00 CRITICAL #12 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x1d9723]
|
memcached.log:2021-02-24T01:40:50.685297-08:00 CRITICAL #13 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x1e6dd4]
|
memcached.log:2021-02-24T01:40:50.685305-08:00 CRITICAL #14 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x19645e]
|
memcached.log:2021-02-24T01:40:50.685312-08:00 CRITICAL #15 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x196b14]
|
memcached.log:2021-02-24T01:40:50.685319-08:00 CRITICAL #16 /opt/couchbase/bin/../lib/libep.so() [0x7fca96fbf000+0x1290f4]
|
memcached.log:2021-02-24T01:40:50.685328-08:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x50bf1]
|
memcached.log:2021-02-24T01:40:50.685337-08:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0xd6e27]
|
memcached.log:2021-02-24T01:40:50.685345-08:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0xd6be8]
|
memcached.log:2021-02-24T01:40:50.685352-08:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0xdc5c0]
|
memcached.log:2021-02-24T01:40:50.685359-08:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0xb0876]
|
memcached.log:2021-02-24T01:40:50.685367-08:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0xb4556]
|
memcached.log:2021-02-24T01:40:50.685373-08:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0xa2389]
|
memcached.log:2021-02-24T01:40:50.685379-08:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x8379d]
|
memcached.log:2021-02-24T01:40:50.685385-08:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x8a7f5]
|
memcached.log:2021-02-24T01:40:50.685390-08:00 CRITICAL #26 /opt/couchbase/bin/memcached() [0x400000+0x8b6bf]
|
memcached.log:2021-02-24T01:40:50.685397-08:00 CRITICAL #27 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7fca949dc000+0xed8e]
|
memcached.log:2021-02-24T01:40:50.685402-08:00 CRITICAL #28 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7fca949dc000+0x17d01]
|
memcached.log:2021-02-24T01:40:50.685408-08:00 CRITICAL #29 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7fca949dc000+0x1854f]
|
memcached.log:2021-02-24T01:40:50.685415-08:00 CRITICAL #30 /opt/couchbase/bin/memcached() [0x400000+0x6bc39]
|
memcached.log:2021-02-24T01:40:50.685422-08:00 CRITICAL #31 /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fca95914000+0x16727]
|
memcached.log:2021-02-24T01:40:50.685428-08:00 CRITICAL #32 /lib64/libpthread.so.0() [0x7fca9315b000+0x7e65]
|
memcached.log:2021-02-24T01:40:50.685460-08:00 CRITICAL #33 /lib64/libc.so.6(clone+0x6d) [0x7fca92d8d000+0xfe88d]
|
This test had passed on 7.0.0-4502. cbcollect_info attached.
Attachments
Issue Links
- duplicates
-
MB-44414 Crash in ActiveDurabilityMonitor::ReplicationChain::hasAcked
- Closed