Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61264

Partition indexes doc count increasing post rebalance

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      Cluster info - 6 index node and 2 kv-query. Build - 7.6.0-2182

      Steps to repro - 

      • Disable shard based rebalance
      • Created a bucket and created one scope and two collections.
      • The first collection had 110k docs , second collection 10k and default collection 10k docs
      • Enable re distribute indexes settings
      • Created partitioned indexes along with other type of indexes(array, primary) on all namespaces.
      • Start continous kv mutations with no increase in no of docs
      • Enable shard based rebalance
      • Triggered a swap rebalance of one node
      • Post the rebalance, there appears to be a count mismatch in item count in the partitioned indexes. 

      Few more relevant timestamps -

      Shard affinity disabled at : 2024-03-22 02:27:49

      Index creation starts at : 2024-03-22 02:39:45

      Shard affinity enabled at : 2024-03-22 02:41:00

      Rebalance is triggered at : 2024-03-22 02:42:48

      Validation fails at : 2024-03-22 02:44:51

      Indexes for which there is item count mismatch

      2024-03-22 02:44:51 | ERROR | MainProcess | test_thread | [tuq_helper._find_differences] Index data sizes differ by more than 0.0 percent before and after rebalance.{'values_changed': {"root['hotel2e5651c9488c4bad9866637bf430a3a0partitioned_index (replica 1)']": {'new_value': 10088, 'old_value': 10000}, "root['hotel0bcda3687ffc4c8aae44946f72b2225cpartitioned_index (replica 2)']": {'new_value': 10156, 'old_value': 10091}, "root['hotel2e5651c9488c4bad9866637bf430a3a0partitioned_index (replica 2)']": {'new_value': 10083, 'old_value': 10000}, "root['hotel0bcda3687ffc4c8aae44946f72b2225cpartitioned_index']": {'new_value': 10149, 'old_value': 10091}, "root['hotele5a026626bf84a24abe9d7b6af8e9a6apartitioned_index']": {'new_value': 110164, 'old_value': 110086}, "root['hotel2e5651c9488c4bad9866637bf430a3a0partitioned_index']": {'new_value': 10065, 'old_value': 10000}, "root['hotel0bcda3687ffc4c8aae44946f72b2225cpartitioned_index (replica 1)']": {'new_value': 10173, 'old_value': 10091}, "root['hotele5a026626bf84a24abe9d7b6af8e9a6apartitioned_index (replica 1)']": {'new_value': 110135, 'old_value': 110086}}} 
        

      Edit

      Logs 

      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.105.122.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.96.198.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.96.230.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.100.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.108.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.109.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.66.zip
      https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-03-22T095612-ns_1%40172.23.97.67.zip

      cc Varun Velamuri 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dhananjay.kshirsagar Dhananjay Kshirsagar
            yash.dodderi Yash Dodderi
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty