Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61516

Planner variation >25% during rebalance with skewed loading of keyspaces

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • Morpheus, 7.6.3
    • 7.6.0
    • secondary-index
    • None
    • debian
    • Untriaged
    • 0
    • Unknown

    Description

      There was a system run for testing the upgrade code path in planner with shard affinity. It was observed that the planner variance was more than 25% during the rebalance. Creating this ticket to investigate the issue further. 

      Build - 7.6.0-2024

      Steps to repro 

      1. Initialize a 8 node cluster with 2 kv:n1ql nodes and 6 index nodes
      2. Create a bucket with 2 scopes and 6 collections and load each keyspace with 10k docs
      3. Load one the collections with 100k docs so that there is skew in data for that keyspace
      4. Create around 13 indexes on each keyspace with num replica ranging from 2-5
      5. Set indexer.settings.rebalance.redistribute_indexes:True
      6. Start continuous mutations on all the keyspaces. 1000 mutations per keyspace
      7. Enable shard based rebalance and swap rebalance one indexer node
      8. swap rebalance two indexer nodes
      9. swap rebalance three indexer nodes
      10. disable shard based rebalance
      11. swap rebalance three indexer nodes
      12. swap rebalance the other three indexer nodes 
      13. Step 7-12 is considered as one cycle. So this cycle was run 20 times

      Logs -
      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.105.12.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.105.13.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.105.14.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.106.180.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.106.183.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.106.189.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.107.20.zip

      https://cb-engineering.s3.amazonaws.com/yash-test-rebalnce/collectinfo-2024-03-27T195550-ns_1%40172.23.107.77.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            amit.kulkarni Amit Kulkarni
            yash.dodderi Yash Dodderi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty