Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
7.6.0
-
EC2 instance c6a.32xlarge
CPUs - 128
RAM - 256GB
-
Triaged
-
Linux x86_64
-
-
0
-
Yes
-
March-June 24
Description
Steps:
- Initialise a 1 node KV cluster
- The cluster is configured with just 2 vbuckets.
- Create a Magma bucket with 250GB memory and start loading data
- Data load consists of doc sizes which are in the range of (2MB - 18MB).
- Loaded about 13,000 docs.
- Rebalanced-in a node and observed the following during rebalance
Since the bucket only has 2vbuckets, during rebalance-in, only 1 vbucket was moving from the source to the destination node. -
Rebalance exited with reason {mover_crashed,
{unexpected_exit,
{'EXIT',<0.20525.1>,
{{{{badmatch,{error,timeout}},
[{mc_client_binary,stats_recv,4,
[{file,"src/mc_client_binary.erl"},
{line,172}]},
{mc_client_binary,stats,4,
[{file,"src/mc_client_binary.erl"},
{line,499}]},
{ns_memcached,do_handle_call,3,
[{file,"src/ns_memcached.erl"},
{line,620}]},
{ns_memcached,worker_loop,3,
[{file,"src/ns_memcached.erl"},
{line,243}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,240}]}]},
{gen_server,call,
['ns_memcached-test_bucket',
{get_dcp_docs_estimate,1,
"replication:ns_1@172.31.30.47->ns_1@ec2-54-88-120-100.compute-1.amazonaws.com:test_bucket"},
300000]}},
{gen_server,call,
[{'janitor_agent-test_bucket',
'ns_1@172.31.30.47'},
{if_rebalance,<0.20500.1>,
{wait_dcp_data_move,
['ns_1@ec2-54-88-120-100.compute-1.amazonaws.com'],
1}},
infinity]}}}}}.
Rebalance Operation Id = 472f8c4c222090b8a956b2c3217f2438
-
Stack backtrace of crashed thread:
#0 /opt/couchbase/bin/memcached() [0x400000+0x835001]
#1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x852ec5]
#2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x85320f]
#3 /lib64/libc.so.6() [0x7f7a77a00000+0x54df0]
#4 /lib64/libc.so.6() [0x7f7a77a00000+0xa154c]
#5 /lib64/libc.so.6(raise+0x16) [0x7f7a77a00000+0x54d46]
#6 /lib64/libc.so.6(abort+0xd3) [0x7f7a77a00000+0x287f3]
#7 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xa89ab]
#8 /opt/couchbase/bin/memcached() [0x400000+0x844fe6]
#9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xb82fa]
#10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xb8365]
#11 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xb85b7]
#12 /opt/couchbase/bin/memcached() [0x400000+0x92c015]
#13 /opt/couchbase/bin/memcached() [0x400000+0xbf5f4]
#14 /opt/couchbase/bin/memcached() [0x400000+0x47e318]
#15 /opt/couchbase/bin/memcached() [0x400000+0x5ab271]
#16 /opt/couchbase/bin/memcached() [0x400000+0x5b37e3]
#17 /opt/couchbase/bin/memcached() [0x400000+0x5b3949]
#18 /opt/couchbase/bin/memcached() [0x400000+0x5b9bb1]
#19 /opt/couchbase/bin/memcached() [0x400000+0x78c76f]
#20 /opt/couchbase/bin/memcached() [0x400000+0x4f77f2]
#21 /opt/couchbase/bin/memcached() [0x400000+0x78a50a]
#22 /opt/couchbase/bin/memcached() [0x400000+0x78d8c3]
#23 /opt/couchbase/bin/memcached() [0x400000+0x92800f]
#24 /opt/couchbase/bin/memcached() [0x400000+0x910b37]
#25 /opt/couchbase/bin/memcached() [0x400000+0x92abaa]
#26 /opt/couchbase/bin/memcached() [0x400000+0x786850]
#27 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7a77e00000+0xe4aa3]
#28 /lib64/libc.so.6() [0x7f7a77a00000+0x9f802]
#29 /lib64/libc.so.6() [0x7f7a77a00000+0x3f450]
- Cb-collect logs are attached.