Details
-
Bug
-
Resolution: Duplicate
-
Major
-
Cheshire-Cat
-
Untriaged
-
Unknown
Description
On a two node vagrant cluster I had done a graceful failover. When a subsequent delta recovery rebalance was done it failed:
The ns_server debug.log on node .101 showed
[user:error,2020-02-24T20:46:35.938Z,ns_1@10.112.210.101:<0.1489.0>:ns_orchestrator:log_rebalance_completion:1446]Rebalance exited with reason {buckets_shutdown_wait_failed, |
[{'ns_1@10.112.210.102', |
{'EXIT', |
{old_buckets_shutdown_wait_failed,
|
["beer-sample","default","default2", |
"magmaBucket"]}}}]}. |
Rebalance Operation Id = fa34adff8f29885dff361cac9047bbae |
The ns_server debug log on node .102 showed
[ns_server:debug,2020-02-24T20:45:15.936Z,ns_1@10.112.210.102:<0.20587.14>:ns_rebalancer:local_buckets_shutdown_loop:192]Waiting until the following old bucket instances are gone: ["beer-sample", |
"default", |
"default2", |
"magmaBucket"] |
<snip>
|
[ns_server:info,2020-02-24T20:46:35.946Z,ns_1@10.112.210.102:rebalance_agent<0.928.0>:rebalance_agent:handle_down:296]Rebalancer process <14531.4056.24> died (reason {buckets_shutdown_wait_failed, |
[{'ns_1@10.112.210.102', |
{'EXIT', |
{old_buckets_shutdown_wait_failed,
|
["beer-sample","default", |
"default2", |
"magmaBucket"]}}}]}). |
The memcached.log.000021.txt file on node .102 showed
2020-02-24T20:44:19.228818+00:00 INFO (magmaBucket) ~VBucket(): vb:1010
|
2020-02-24T20:44:19.228855+00:00 INFO (magmaBucket) ~VBucket(): vb:1014
|
2020-02-24T20:44:19.228884+00:00 INFO (magmaBucket) ~VBucket(): vb:1018
|
2020-02-24T20:44:19.228926+00:00 INFO (magmaBucket) ~VBucket(): vb:1022
|
2020-02-24T20:44:24.530928+00:00 WARNING (No Engine) Slow runtime for 'DurabilityTimeoutVisitor on vb:516' on thread nonIO_worker_1: 109 ms
|
2020-02-24T20:44:24.572929+00:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-10990). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/649e0b96-36d1-d37c-0f44925a-6be26824.dmp before terminating.
|
2020-02-24T20:44:24.572964+00:00 CRITICAL Stack backtrace of crashed thread:
|
2020-02-24T20:44:24.579445+00:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x13bd7d]
|
2020-02-24T20:44:24.579469+00:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x152f7e]
|
2020-02-24T20:44:24.579479+00:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x153294]
|
2020-02-24T20:44:24.579486+00:00 CRITICAL /lib64/libpthread.so.0() [0x7f104857c000+0xf130]
|
2020-02-24T20:44:24.579506+00:00 CRITICAL /lib64/libc.so.6(gsignal+0x39) [0x7f10481bb000+0x355c9]
|
2020-02-24T20:44:24.579552+00:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f10481bb000+0x36cd8]
|
2020-02-24T20:44:24.579556+00:00 CRITICAL /opt/couchbase/bin/../lib/libjemalloc.so.2(je_dallocx+0x72c) [0x7f104a437000+0x202ec]
|
2020-02-24T20:44:24.579565+00:00 CRITICAL /opt/couchbase/bin/memcached(cb_free+0x1a) [0x400000+0x116c4a]
|
2020-02-24T20:44:24.579648+00:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1048cb1000+0x8d192]
|
2020-02-24T20:44:24.579656+00:00 CRITICAL /lib64/libpthread.so.0() [0x7f104857c000+0x7bf2]
|
2020-02-24T20:44:24.579661+00:00 CRITICAL /lib64/libpthread.so.0() [0x7f104857c000+0x7e01]
|
2020-02-24T20:44:24.579692+00:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f10481bb000+0xf61ad]
|
Attachments
Issue Links
- duplicates
-
MB-37998 [Magma] Coredump seen during rebalance in out with hard failover
- Closed