Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48918

[FTS] Rebalance not distributing indexes evenly across nodes

    XMLWordPrintable

Details

    • 1

    Description

      Discovered this behaviour during an iteration of the experiment mentioned in MB-47441

      Setup:

      • Cluster setup: 2 nodes, one with FTS and KV services enabled and another with only FTS enabled(which is added later followed by rebalance).
      • FTS quota - 17GB, KV - 8GB
      • System specific details - each node has 8 cores, 32 GB RAM and 128 GB storage with Centos 64-bit OS.
      • Each index has 1 partition and indexes a field having 400 chars and 2 numeric fields; each document has 1000 chars and 2 numeric fields.

      Details:

      1. Indexes were created as per the procedure mentioned in MB-47441 on the first node({FTS,KV}), till the count reached 1000.
      2. The second node ({FTS}) was later added to the cluster and a rebalance was performed. The procedure completed successful within 2.5 to 5 mins. 
      3. On physically checking the new node, it was found there were no actual pindex files in it and also noticed that the planPIndexes hadn't changed (indexes' UUID was that of the first node). 
      4. Repeated this in a scaled down scenario where around 100 indexes were created, and on rebalance the same observations were made.
      5. Also, while adding a new index after the rebalance was completed, it was found that this 101th index resides in the new node. Note: Currently, verifying this in the 1000 indexes scenario.
      6. Since on a rebalance there is a visible imbalance in the index distribution across cluster, it looks like there could be a scope of improvement in this area.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thejas.orkombu Thejas Orkombu
              thejas.orkombu Thejas Orkombu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty