Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Couchbase Enterprise Build 7.0.0-2908
-
Untriaged
-
Centos 64-bit
-
-
1
-
No
Description
Script to Repo
./testrunner -i /tmp/durability_volume.ini rerun=False,skip_validations=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,quota_percent=80,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE
|
Steps to Repro:
1. Create a 5 node cluster
2020-08-28 15:19:34,111 | test | INFO | pool-26-thread-7 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.105.211 | kv | Cluster node |
| 172.23.105.212 | None | <--- IN — |
| 172.23.105.213 | None | <--- IN — |
| 172.23.105.215 | None | <--- IN — |
| 172.23.105.217 | None | <--- IN — |
-----------------------++-------------
2. Initial data load to buckets
2020-08-28 15:31:48,332 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-----------------+-----------------------------------------------------+----------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-----------------+-----------------------------------------------------+----------
| bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 188875200 | 233891705 |
| bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 188473008 | 246609666 |
| default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 476117456 | 426023731 |
-----------------+-----------------------------------------------------+----------
3. Failover node (.217) while data load load in parallel
2020-08-28 15:34:09,966 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:129] 1 nodes failed over as expected in 0.0520000457764 seconds
4. Delta Recover the node and rebalance
2020-08-28 15:36:20,309 | test | INFO | pool-26-thread-19 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.105.215 | kv | Cluster node |
| 172.23.105.212 | kv | Cluster node |
| 172.23.105.213 | kv | Cluster node |
| 172.23.105.217 | kv | Cluster node |
| 172.23.105.211 | kv | Cluster node |
-----------------------++-------------
rebalance fails with coredump on .217
ST of crash on .217 0f2a1af9-8184-443e-2d9cf2ba-88e2dfe4.dmp
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007fbb27886207 in raise () from /lib64/libc.so.6
|
#0 0x00007fbb27886207 in raise () from /lib64/libc.so.6
|
#1 0x00007fbb278878f8 in abort () from /lib64/libc.so.6
|
#2 0x00007fbb283e3195 in _gnu_cxx::_verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#3 0x000000000054ebb2 in backtrace_terminate_handler() ()
|
#4 0x00007fbb283e0f86 in _cxxabiv1::_terminate(void ()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#5 0x00007fbb283e0fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#6 0x00007fbb283e1213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#7 0x00007fbb2bb77188 in gsl::fail_fast_assert(bool, char const*) () from /opt/couchbase/bin/../lib/libep.so
|
#8 0x00007fbb2bcbb7a2 in VBucket::notifyClientOfSyncWriteComplete(void const*, ENGINE_ERROR_CODE) () from /opt/couchbase/bin/../lib/libep.so
|
#9 0x00007fbb2bbe544c in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
|
#10 0x00007fbb2bbe5683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
|
#11 0x00007fbb2bcb68c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
|
#12 0x00007fbb2bbec68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
|
#13 0x00007fbb2bc51583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
|
#14 0x00007fbb2bb647df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
|
#15 0x00007fbb2a5e7777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
|
#16 0x00007fbb27c24dd5 in start_thread () from /lib64/libpthread.so.0
|
#17 0x00007fbb2794dead in clone () from /lib64/libc.so.6
|
Attached cbcollect logs.