Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62805

Aborted rebalance leads to increased partition count and uneven distribution

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.6.4
    • 7.2.5, 7.6.2
    • fts
    •  7.2.5 build 7599

    Description

      Create 3 node cluster

      1. Failover one node
        1. Upgrade the node to 7.6.2-3721
        2. Rebalance(remove out the node) and stop rebalance without it completing
        3. Add node back, rebalance

      The partition count increases for no reason. 18 parition becomes 20 and 3 partitions become 5. This happens for both active and replica parition

      Partitions before:

      Actives:
      		 
              172.23.122.68:8094 : 8
      		 
              172.23.107.56:8094 : 8
      		 
              172.23.120.23:8094 : 8
      		 
      Replicas:
      		 
              172.23.122.68:8094 : 8
      		 
              172.23.107.56:8094 : 8
      		 
              172.23.120.23:8094 : 8
      		 
      Actual number of index partitions in cluster: 48
      		 
      Expected number of index partitions in cluster: 48
      		 
      Indexes: 3
      		 
              n2-copy-copy :: maxPartitionsPerPIndex: 342, indexPartitions: 3, numReplicas: 1
      		 
              n2 :: maxPartitionsPerPIndex: 57, indexPartitions: 18, numReplicas: 1
      		 
              n2-copy :: maxPartitionsPerPIndex: 342, indexPartitions: 3, numReplicas: 1Index actives distribution:
      		 
              Index: n2-copy
      		 
                      172.23.122.68:8094 : 1
      		 
                      172.23.107.56:8094 : 1
      		 
                      172.23.120.23:8094 : 1
      		 
              Index: n2
      		 
                      172.23.107.56:8094 : 6
      		 
                      172.23.120.23:8094 : 6
      		 
                      172.23.122.68:8094 : 6
      		 
              Index: n2-copy-copy
      		 
                      172.23.120.23:8094 : 1
      		 
                      172.23.122.68:8094 : 1
      		 
                      172.23.107.56:8094 : 1
      		 
      Index replicas distribution:
      		 
              Index: n2-copy
      		 
                      172.23.107.56:8094 : 1
      		 
                      172.23.120.23:8094 : 1
      		 
                      172.23.122.68:8094 : 1
      		 
              Index: n2
      		 
                      172.23.120.23:8094 : 6
      		 
                      172.23.122.68:8094 : 6
      		 
                      172.23.107.56:8094 : 6
      		 
              Index: n2-copy-copy
      		 
                      172.23.107.56:8094 : 1
      		 
                      172.23.120.23:8094 : 1
      		 
                      172.23.122.68:8094 : 1

      Skew after:

       

      Actives:
      		 
              172.23.122.68:8094 : 10
      		 
              172.23.107.56:8094 : 10
      		 
              172.23.120.23:8094 : 10
      		 
      Replicas:
      		 
              172.23.120.23:8094 : 9
      		 
              172.23.122.68:8094 : 9
      		 
              172.23.107.56:8094 : 8
      		 
      Actual number of index partitions in cluster: 56
      		 
      Expected number of index partitions in cluster: 48
      		 
      Indexes: 3
      		 
              n2 :: maxPartitionsPerPIndex: 57, indexPartitions: 18, numReplicas: 1
      		 
              n2-copy :: maxPartitionsPerPIndex: 342, indexPartitions: 3, numReplicas: 1
      		 
              n2-copy-copy :: maxPartitionsPerPIndex: 342, indexPartitions: 3, numReplicas: 1Index actives distribution:
      		 
              Index: n2-copy
      		 
                      172.23.107.56:8094 : 2
      		 
                      172.23.122.68:8094 : 2
      		 
                      172.23.120.23:8094 : 1
      		 
              Index: n2-copy-copy
      		 
                      172.23.122.68:8094 : 2
      		 
                      172.23.120.23:8094 : 2
      		 
                      172.23.107.56:8094 : 1
      		 
              Index: n2
      		 
                      172.23.120.23:8094 : 7
      		 
                      172.23.122.68:8094 : 6
      		 
                      172.23.107.56:8094 : 7
      		 
      Index replicas distribution:
      		 
              Index: n2-copy-copy
      		 
                      172.23.107.56:8094 : 1
      		 
                      172.23.120.23:8094 : 1
      		 
                      172.23.122.68:8094 : 2
      		 
              Index: n2
      		 
                      172.23.122.68:8094 : 6
      		 
                      172.23.120.23:8094 : 7
      		 
                      172.23.107.56:8094 : 6
      		 
              Index: n2-copy
      		 
                      172.23.122.68:8094 : 1
      		 
                      172.23.120.23:8094 : 1
      		 
                      172.23.107.56:8094 : 1

       

       

      Attachments

        Issue Links

          depends on

          Improvement - An improvement or enhancement to an existing feature or task. MB-62841 Allow Search service greater control in picking rebalance orchestrator.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarthak.dua Sarthak Dua
              sarthak.dua Sarthak Dua
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty