Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43273

[FTS] Investigate slowness of rebalance time in CC

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • fts
    • 1

    Description

      There is a slowness observed in rebalance time when the maxConcurrentMovesPerNode is set. No significant decrease in time is observed. 

      FTS RAM Quota :102400 MB , Rebalance test run : Rebalance -in test
      Parameter value: on 6.6.1 - toy on CC
      Default = 1 27.3 min 45.4 min
      maxConcurrentMovesPerNode = 10 11.9 min 42.0 min
      Index size  ~ 260000 MB ~ 190000 MB
      Time taken by index to complete indexing(with parameter = 1) 53 mins 45.5 min
      Time taken by index to complete indexing(with parameter= 10) 61 mins 46.5 min

      Logs for Toy : 
      For test where maxConcurrentMovesPerNode = 1 (default) : 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-571/172.23.99.211.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-571/172.23.99.38.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-571/172.23.99.39.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-571/172.23.99.40.zip

      For test where maxConcurrentMovesPerNode =  10 :
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-570/172.23.99.211.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-570/172.23.99.38.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-570/172.23.99.39.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_dev-570/172.23.99.40.zip

      Logs for CC :
      For test where maxConcurrentMovesPerNode = 1 (default) : 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-29/172.23.99.211.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-29/172.23.99.38.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-29/172.23.99.40.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-29/172.23.99.39.zip 

      For test where maxConcurrentMovesPerNode =  10 : 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-44/172.23.99.211.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-44/172.23.99.38.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-44/172.23.99.39.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas_rebalance-44/172.23.99.40.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-44877 [System Test] : FTS rebalance stuck for over 17hrs

          • Test Blocker - QE-related tests can't proceed
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-44725 [System Test]service_rebalance_failed,fts: rebalance_failed,inactivity_timeout

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jyotsna.nayak Jyotsna Nayak
              jyotsna.nayak Jyotsna Nayak
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty