Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
None
-
Triaged
-
1
-
Unknown
-
KV 2022-Jan
Description
Using vagrant/centos7 I configured a two node cluster running couchbase-server-enterprise-7.0.0-5302-centos7.x86_64.rpm.
I then did a graceful failover of node1 and then stopped couchbase-server on node1. I then installed "rpm -U couchbase-server-enterprise-7.1.0-2029-centos7.x86_64.rpm" on node1. Once couchbase came up I attempted a delta-recovery rebalance which failed:
"completionMessage": "Rebalance exited with reason {prepare_delta_recovery_failed,\"travel-sample\",\n {error,\n {failed_nodes,\n [{'ns_1@10.112.213.102',{error,aborted}}]}}}."
|
Collected bundles from both nodes
https://s3.amazonaws.com/cb-engineering/stevewatanabe/13Jan2022-prepareDeltaRecoveryFailure/collectinfo-2022-01-13T205804-ns_1%4010.112.213.101.zip
https://s3.amazonaws.com/cb-engineering/stevewatanabe/13Jan2022-prepareDeltaRecoveryFailure/collectinfo-2022-01-13T205804-ns_1%4010.112.213.102.zip
From the ns_server.debug.log for the 7.1 node here's the failure
=========================CRASH REPORT=========================
|
crasher:
|
initial call: misc:'-spawn_monitor/1-fun-0-'/0
|
pid: <0.5519.0>
|
registered_name: []
|
exception exit: {{{nocatch,{error,closed}},
|
[{mc_binary,recv_with_data,4,
|
[{file,"src/mc_binary.erl"},{line,41}]},
|
{mc_binary,quick_active_recv,3,
|
[{file,"src/mc_binary.erl"},{line,48}]},
|
{mc_binary,quick_stats_loop_enter,5,
|
[{file,"src/mc_binary.erl"},{line,100}]},
|
{mc_binary,quick_stats,5,
|
[{file,"src/mc_binary.erl"},{line,85}]},
|
{ns_memcached,do_handle_call,3,
|
[{file,"src/ns_memcached.erl"},{line,541}]},
|
{ns_memcached,worker_loop,3,
|
[{file,"src/ns_memcached.erl"},{line,225}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,226}]}]},
|
{gen_server,call,
|
[{'ns_memcached-travel-sample','ns_1@10.112.213.102'},
|
list_vbuckets,300000]}}
|
in function gen_server:call/3 (gen_server.erl, line 247)
|
in call from ns_memcached:do_call/4 (src/ns_memcached.erl, line 1406)
|
in call from rebalance_agent:prepare_vbuckets/2 (src/rebalance_agent.erl, line 399)
|
in call from rebalance_agent:prepare_delta_recovery_bucket_job/3 (src/rebalance_agent.erl, line 233)
|
in call from rebalance_agent:'-run_async_job/2-fun-0-'/1 (src/rebalance_agent.erl, line 782)
|
in call from gen_server2:'-start_job/2-fun-0-'/3 (src/gen_server2.erl, line 379)
|
Looking at the memcached.log
2022-01-13T12:55:08.727422-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2022-01-13T12:55:08.894654-08:00 INFO ---------- Opening logfile:
|
2022-01-13T12:55:08.895863-08:00 INFO Couchbase version 7.1.0-2029 starting.
|
2022-01-13T12:55:08.895896-08:00 INFO Process identifier: 13513
|
2022-01-13T12:55:08.895900-08:00 INFO Development asserts enabled
|
2022-01-13T12:55:08.895929-08:00 INFO recalculate_max_connections: {"engine_fds":133982,"max_connections":65000,"max_fds":200000,"system_connections":5000}
|
2022-01-13T12:55:08.896049-08:00 INFO Breakpad enabled. Minidumps will be written to '/opt/couchbase/var/lib/couchbase/crash'
|
2022-01-13T12:55:08.896071-08:00 CRITICAL Detected previous crash
|
2022-01-13T12:55:08.896165-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2029). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/24cfe0c5-5531-4ad4-dc4cc9a9-52e042b1.dmp before terminating.
|
2022-01-13T12:55:08.896188-08:00 CRITICAL Stack backtrace of crashed thread:
|
2022-01-13T12:55:08.896192-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x725298]
|
2022-01-13T12:55:08.896194-08:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x77697a]
|
2022-01-13T12:55:08.896196-08:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x776cb8]
|
2022-01-13T12:55:08.896198-08:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f515eb82000+0xf630]
|
2022-01-13T12:55:08.896199-08:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f515c3cc000+0x36387]
|
2022-01-13T12:55:08.896201-08:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f515c3cc000+0x37a78]
|
2022-01-13T12:55:08.896203-08:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f515ccb4000+0x9963c]
|
2022-01-13T12:55:08.896205-08:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x7338fb]
|
2022-01-13T12:55:08.896206-08:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f515ccb4000+0xa48f6]
|
2022-01-13T12:55:08.896208-08:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f515ccb4000+0xa4961]
|
2022-01-13T12:55:08.896209-08:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f515ccb4000+0xa4bf4]
|
2022-01-13T12:55:08.896211-08:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xa884e]
|
2022-01-13T12:55:08.896213-08:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0xba58a]
|
2022-01-13T12:55:08.896215-08:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x34afc5]
|
2022-01-13T12:55:08.896216-08:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x353221]
|
2022-01-13T12:55:08.896218-08:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x6a2a12]
|
2022-01-13T12:55:08.896219-08:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x69fc25]
|
2022-01-13T12:55:08.896221-08:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x7f62a0]
|
2022-01-13T12:55:08.896223-08:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x7de05a]
|
2022-01-13T12:55:08.896224-08:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x7f9259]
|
2022-01-13T12:55:08.896226-08:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x69f8b4]
|
2022-01-13T12:55:08.896227-08:00 CRITICAL #21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f515ccb4000+0xcdd40]
|
2022-01-13T12:55:08.896229-08:00 CRITICAL #22 /lib64/libpthread.so.0() [0x7f515eb82000+0x7ea5]
|
2022-01-13T12:55:08.896231-08:00 CRITICAL #23 /lib64/libc.so.6(clone+0x6d) [0x7f515c3cc000+0xfe96d]
|