Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.0
-
7.2.0-5150
-
Untriaged
-
Centos 64-bit
-
-
0
-
Unknown
-
KV 2023-2
Description
Build: 7.2.0-5150
Scenario:
- 4 node cluster
- Magma bucket with replica=3, retention_bytes=1000000000,seconds=86400
- Perform cont. dedupe mutations and trigger rebalance_in for 2 nodes in parallel
+----------------+----------+-----------------------+---------------+--------------+| Nodes | Services | Version | CPU | Status |+----------------+----------+-----------------------+---------------+--------------+| 172.23.105.212 | kv | 7.2.0-5150-enterprise | 96.6901059859 | Cluster node || 172.23.105.155 | kv | 7.2.0-5150-enterprise | 94.4061862745 | Cluster node || 172.23.105.213 | kv | 7.2.0-5150-enterprise | 94.4647571316 | Cluster node || 172.23.105.211 | kv | 7.2.0-5150-enterprise | 98.2583104595 | Cluster node || 172.23.105.244 | None | | | <--- IN --- || 172.23.105.245 | None | | | <--- IN --- |+----------------+----------+-----------------------+---------------+--------------+
Observation:
Memcached crashed on node .212 resulting in rebalance failure.
Rebalance exited with reason {{badmatch,
|
{error,
|
{failed_nodes,['ns_1@172.23.105.212']}}},
|
[{ns_janitor,cleanup_apply_config_body,4,
|
[{file,"src/ns_janitor.erl"},{line,295}]},
|
{ns_janitor,'-cleanup_apply_config/4-fun-0-',
|
4,
|
[{file,"src/ns_janitor.erl"},{line,215}]},
|
{async,'-async_init/4-fun-1-',3,
|
[{file,"src/async.erl"},{line,191}]}]}.
|
Rebalance Operation Id = 2d98d03e8da97bd0d1df294eacf3474e
|
|
|
Node :: ns_1@172.23.105.212
|
Service 'memcached' exited with status 134. Restarting. Messages:
|
#16 /opt/couchbase/bin/memcached() [0x400000+0x331384]
|
#17 /opt/couchbase/bin/memcached() [0x400000+0x334052]
|
#18 /opt/couchbase/bin/memcached() [0x400000+0x2fdfb8]
|
#19 /opt/couchbase/bin/memcached() [0x400000+0x4eb5c8]
|
#20 /opt/couchbase/bin/memcached() [0x400000+0x4ebe85]
|
#21 /opt/couchbase/bin/memcached() [0x400000+0x4dcf72]
|
#22 /opt/couchbase/bin/memcached() [0x400000+0x4e3517]
|
#23 /opt/couchbase/bin/memcached() [0x400000+0x4e6a45]
|
#24 /opt/couchbase/bin/memcached() [0x400000+0x6a3279]
|
#25 /opt/couchbase/bin/memcached() [0x400000+0x69c93a]
|
#26 /opt/couchbase/bin/memcached() [0x400000+0x6a462e]
|
#27 /opt/couchbase/bin/memcached() [0x400000+0x801c10]
|
#28 /opt/couchbase/bin/memcached() [0x400000+0x7ec6aa]
|
#29 /opt/couchbase/bin/memcached() [0x400000+0x804bc9]
|
#30 /opt/couchbase/bin/memcached() [0x400000+0x69c634]
|
#31 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f894672c000+0xcdd40]
|
#32 /lib64/libpthread.so.0() [0x7f89485fa000+0x7dd5]
|
#33 /lib64/libc.so.6(clone+0x6d) [0x7f8945e45000+0xfdead]
|
Memcached gdb backtrace: memcahced_bt.txt