Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11279

UPR-based rebalance (3 -> 4) is extremely slow due to occasionally slow backfilling

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • 3.0.0-757

      Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2630 (24 vCPU)
      Memory = 64 GB
      Disk = RAID 10 HDD

    Description

      KV-only rebalance test, 3 -> 4 nodes, 1 bucket x 100M x 2KB, 10K mixed ops/sec

      We recently enabled UPR-based rebalance by default. As result we are > 20x slower now (20 minutes vs. 460 minutes).

      Most importantly there is inconsistency in backfilling speed, some movements take 20-30 minutes:

      http://cbmonitor.sc.couchbase.com/reports/movements/?filename=0595c86ba9494c6da303c870c25c641a [the diagram is interactive]

      The way it was before:

      http://cbmonitor.sc.couchbase.com/reports/movements/?filename=78e473492d2e40e88cea8dc88d9274a8

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavelpaulau Pavel Paulau (Inactive)
            pavelpaulau Pavel Paulau (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty