Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
Elixir
-
7.2.0-2132/serverless
-
Untriaged
-
Centos 64-bit
-
1
-
No
Description
Steps to Repro
1. Create a 5 node cluster.
2022-09-23 19:12:11,303 | test | INFO | MainThread | [table_view:display:72] Cluster statistics
|
+----------------+---------+----------+--------+-----------+-----------+----------------------+-------------------+------------------------------------+
|
| Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config |
|
+----------------+---------+----------+--------+-----------+-----------+----------------------+-------------------+------------------------------------+
|
| 172.23.108.251 | Group 1 | kv | 0.3049 | 11.45 GiB | 10.69 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-2132-enterprise / serverless |
|
| 172.23.105.36 | Group 1 | kv | 0.4898 | 11.45 GiB | 10.69 GiB | 39.41 MiB / 3.50 GiB | 0 / 0 | 7.2.0-2132-enterprise / serverless |
|
| 172.23.105.33 | Group 1 | kv | 1.2384 | 11.45 GiB | 10.69 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-2132-enterprise / serverless |
|
| 172.23.104.76 | Group 1 | kv | 1.1269 | 11.45 GiB | 10.58 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-2132-enterprise / serverless |
|
| 172.23.121.79 | Group 1 | kv | 1.1837 | 11.45 GiB | 10.68 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-2132-enterprise / serverless |
|
+----------------+---------+----------+--------+-----------+-----------+----------------------+-------------------+------------------------------------+
|
2. Create 10 magma bucket of 256 MB and push these buckets to 1 DGM.
3. Add a node(172.23.123.5) , remove a node(172.23.104.76) and start a swap rebalace.
After swap rebalance of some 7 buckets successfully, I noticed following messages during 8th bucket rebalance and rebalance failed.
172.23.123.5 10:34:33 AM 25 Sep, 2022
Service 'memcached' exited with status 134. Restarting. Messages:
|
2022-09-23T19:10:44.977099-07:00 WARNING Unknown key "enforce_tenant_limits_enabled" in config ignored.
|
2022-09-25T10:34:32.914842-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2022-09-25T10:34:32.915100-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'compactions->find(lowStr) == compactions->end()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:1137
|
terminate called after throwing an instance of 'gsl::fail_fast'
|
what(): GSL: Precondition failure: 'compactions->find(lowStr) == compactions->end()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:1137
|
172.23.104.76 10:34:33 AM 25 Sep, 2022
Worker <0.4924.1993> (for action {move,
|
{26,
|
['ns_1@172.23.105.36',
|
'ns_1@172.23.104.76',
|
'ns_1@172.23.105.33'],
|
['ns_1@172.23.105.36',
|
'ns_1@172.23.123.5',
|
'ns_1@172.23.105.33'],
|
[]}}) exited with reason {unexpected_exit,
|
{'EXIT',
|
<0.31864.1992>,
|
{{dcp_wait_for_data_move_failed,
|
"default8",
|
26,
|
'ns_1@172.23.105.36',
|
['ns_1@172.23.123.5',
|
'ns_1@172.23.105.33'],
|
{error,
|
no_stats_for_this_vbucket}},
|
[{ns_single_vbucket_mover,
|
'-wait_dcp_data_move/5-fun-0-',
|
5,
|
[{file,
|
"src/ns_single_vbucket_mover.erl"},
|
{line,
|
451}]},
|
{proc_lib,
|
init_p,3,
|
[{file,
|
"proc_lib.erl"},
|
{line,
|
211}]}]}}}
|
172.23.104.76 10:34:33 AM 25 Sep, 2022
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.31864.1992>,
|
{{dcp_wait_for_data_move_failed,"default8",
|
26,'ns_1@172.23.105.36',
|
['ns_1@172.23.123.5','ns_1@172.23.105.33'],
|
{error,no_stats_for_this_vbucket}},
|
[{ns_single_vbucket_mover,
|
'-wait_dcp_data_move/5-fun-0-',5,
|
[{file,"src/ns_single_vbucket_mover.erl"},
|
{line,451}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,211}]}]}}}}.
|
Rebalance Operation Id = 4b6a2acdcb56b546137190843cf52cf7
|
cbcollect_info attached.
Attachments
For Gerrit Dashboard: MB-53818 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
180596,3 | MB-53818 magma: Don't enforce exclusive range for sstables in L0 | master | magma | Status: MERGED | +2 | +1 |
180708,4 | MB-53818 lsm: Add test to verify multiple concurrent L0 local compactions | master | magma | Status: MERGED | +2 | +1 |