Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.6.0
-
None
-
Untriaged
-
0
-
Unknown
Description
Cluster info - 6 index node and 2 kv-query. Build - 7.6.0-2182
Steps to repro -
- Disable shard based rebalance
- Created a bucket and created one scope and two collections.
- The first collection had 110k docs , second collection 10k and default collection 10k docs
- Enable re distribute indexes settings
- Created partitioned indexes along with other type of indexes(array, primary) on all namespaces.
- Start continous kv mutations with no increase in no of docs
- Enable shard based rebalance
- Triggered a swap rebalance of one node
- Post the rebalance, there appears to be a count mismatch in item count in the partitioned indexes.
Few more relevant timestamps -
Shard affinity disabled at : 2024-03-22 02:27:49
Index creation starts at : 2024-03-22 02:39:45
Shard affinity enabled at : 2024-03-22 02:41:00
Rebalance is triggered at : 2024-03-22 02:42:48
Validation fails at : 2024-03-22 02:44:51
Indexes for which there is item count mismatch
2024-03-22 02:44:51 | ERROR | MainProcess | test_thread | [tuq_helper._find_differences] Index data sizes differ by more than 0.0 percent before and after rebalance.{'values_changed': {"root['hotel2e5651c9488c4bad9866637bf430a3a0partitioned_index (replica 1)']": {'new_value': 10088, 'old_value': 10000}, "root['hotel0bcda3687ffc4c8aae44946f72b2225cpartitioned_index (replica 2)']": {'new_value': 10156, 'old_value': 10091}, "root['hotel2e5651c9488c4bad9866637bf430a3a0partitioned_index (replica 2)']": {'new_value': 10083, 'old_value': 10000}, "root['hotel0bcda3687ffc4c8aae44946f72b2225cpartitioned_index']": {'new_value': 10149, 'old_value': 10091}, "root['hotele5a026626bf84a24abe9d7b6af8e9a6apartitioned_index']": {'new_value': 110164, 'old_value': 110086}, "root['hotel2e5651c9488c4bad9866637bf430a3a0partitioned_index']": {'new_value': 10065, 'old_value': 10000}, "root['hotel0bcda3687ffc4c8aae44946f72b2225cpartitioned_index (replica 1)']": {'new_value': 10173, 'old_value': 10091}, "root['hotele5a026626bf84a24abe9d7b6af8e9a6apartitioned_index (replica 1)']": {'new_value': 110135, 'old_value': 110086}}} |
|
Edit
Logs
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.105.122.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.96.198.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.96.230.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.100.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.108.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.109.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.66.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.67.zip