Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.0
-
Untriaged
-
Linux x86_64
-
-
0
-
Unknown
-
KV 2023-2, KV 2023-4
Description
Steps:
- 5 node KV cluster, 3 buckets
+----------------+-----------------+-----------+-----------+-------------------+
| Node | CPU_utilization | Mem_total | Mem_free | Active / Replica |
+----------------+-----------------+-----------+-----------+-------------------+
| 172.23.107.76 | 4.73959062031 | 23.36 GiB | 21.39 GiB | 1665900 / 3318800 |
| 172.23.107.221 | 9.32815404845 | 23.36 GiB | 21.42 GiB | 1661950 / 3320400 |
| 172.23.107.240 | 4.86830349442 | 23.36 GiB | 21.52 GiB | 1649350 / 3319450 |
| 172.23.107.126 | 4.3131245981 | 23.36 GiB | 21.46 GiB | 1656500 / 3301550 |
| 172.23.107.237 | 5.2971514875 | 23.36 GiB | 21.46 GiB | 1656300 / 3319800 |
+----------------+-----------------+-----------+-----------+-------------------++---------+-----------+-----------------+----------+---------+-----------+------------+------------+---------------+
| Bucket | Type | Storage Backend | Replicas | Items | RAM Quota | RAM Used | Disk Used | ARR |
+---------+-----------+-----------------+----------+---------+-----------+------------+------------+---------------+
| bucket1 | couchbase | couchstore | 2 | 100000 | 9.77 GiB | 320.08 MiB | 235.94 MiB | 100 |
| bucket2 | couchbase | magma | 2 | 50000 | 4.88 GiB | 464.66 MiB | 376.77 MiB | 100 |
| default | couchbase | magma | 2 | 8140000 | 2.50 GiB | 1.61 GiB | 17.68 GiB | 20.5636609337 |
+---------+-----------+-----------------+----------+---------+-----------+------------+------------+---------------+
- Load docs to all bucket + CDC load on magma buckets
- Perform graceful FO of node "172.23.107.240"
- Start dedupe load on the bucket + perform delta recovery and trigger rebalance
Graceful failover completed successfully.
Rebalance Operation Id = 592da811600f15f5f4bcff8beafad703
Observation:
Rebalance of magma bucket 'default' got stuck
Note: I was able to reproduce this consistently on the same build(2/2) runs
TAF test:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini -p rerun=False,get-cbcollect-info=False,skip_cluster_reset=True,upgrade_version=7.2.0-5318 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability,durability=MAJORITY,bucket_spec=magma_dgm.20_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,skip_validations=False,default_history_retention_for_collections=false,bucket_history_retention_seconds=86400,bucket_history_retention_bytes=0,dedupe_update_itrs=10000,disk_optimized_thread_settings=True,get-cbcollect-info=True,autoCompactionDefined=true'
|