Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24713

Duration of rebalance with ephemeral buckets doubled in recent Spock builds

    XMLWordPrintable

Details

    Description

      Build 5.0.0-3042. Last known good build is 5.0.0-2909. Rebalance was broken (seg. faults) after that for two weeks.

      Setup:

      • 9 -> 10 nodes
      • 1 ephemeral bucket (NRU)
      • 500M documents
      • 50K mixed ops/sec

      Rebalance time increased from ~3 minutes to ~7 minutes. It takes about 8 minutes to rebalance the same cluster with Couchbase bucket.

      Graphs:
      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_500-2909__rebalance_b768&snapshot=titan_500-3042_rebalance_4d28

      See also http://showfast.sc.couchbase.com/#/timeline/Linux/reb/kv/all

      Attachments

        1. buffered_trace.json.gz
          1.92 MB
          Oliver Downard
        2. fast_trace.json.gz
          1.44 MB
          Oliver Downard
        3. Screen Shot 2017-06-26 at 09.17.49.png
          716 kB
          Oliver Downard
        4. Screen Shot 2017-06-26 at 09.20.20.png
          411 kB
          Oliver Downard
        5. Screen Shot 2017-07-05 at 13.55.18.png
          35 kB
          Oliver Downard
        6. Screen Shot 2017-07-05 at 14.00.50.png
          39 kB
          Oliver Downard
        7. Screen Shot 2017-07-12 at 10.05.17 AM.png
          93 kB
          Poonam Dhavale
        8. Screen Shot 2017-07-12 at 10.05.17 AM.png
          93 kB
          Poonam Dhavale
        9. Screenshot at Jul 05 14-19-16.png
          163 kB
          Oliver Downard
        10. Screenshot at Jul 05 14-22-49.png
          187 kB
          Oliver Downard
        11. slow_trace.json.gz
          1.29 MB
          Oliver Downard

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-24681 Ephemeral Backfill Memory Management: Finish one backfill completely before proceeding to other backfills

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty