Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11323

Rebalance is 2-3 times slower due to actions not related to backfilling or persistence

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 3.0
    • 3.0
    • ns_server
    • Security Level: Public
    • Build 3.0.0-773

      Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2630 (24 vCPU)
      Memory = 64 GB
      Disk = RAID 10 HDD

    Description

      Rebalance test, 3 -> 4 nodes, 1 bucket x 100M x 2KB, 10K mixed ops/sec

      We had a huge regression in ep-engine which was fixed recently by Mike.
      But rebalance is still slower than it used to be (~20minutes vs. 50 minutes),

      Master events indicate some unusual slowness in vbuckets movements. E.g., now extra time is spent in:
      takeoverStarted -> takeoverEnded
      vbucketStateChange (the last one)
      vbucketStateChange -> vbucketMoverTerminate

      This is not quite fair comparison with TAP-based rebalance but on average each individual movement is 3-4x slower now.

      You can try to play with this profile:
      http://cbmonitor.sc.couchbase.com/reports/movements/?filename=2b20aa8fee5d480fb85ba9e1b46fefe6

      Similar diagram for TAP:
      http://cbmonitor.sc.couchbase.com/reports/movements/?filename=65befd4031914ba1ae811932ce41def0

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            pavelpaulau Pavel Paulau (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty