Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7676

Rebalance hangs when [presumably] data compaction takes place

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Physical machines; 128GB, 12 cores, 2 SATA drives; CentOS 5.8
      build 2.0.1-147

      Description

      Rebalance (in 3->4) + views (1 ddoc x 3 views) + front-end workload (2K ops/sec per node, 200 stake=ok queries/s).

      Master events for both 2.0 and 2.0.1 are attached.

      2.0 diags and debug stats:
      http://172.23.96.10:8080/job/apollo-views/17/artifact/

      2.0.1 diags and debug stats:
      http://172.23.96.10:8080/job/apollo-views/18/artifact/

      Charts are rather interesting imho, even for 2.0 there are periods of slowness. And it definitively correlates with data (bucket) compaction.

      1. master_events_2.0.1.log
        5.35 MB
        Pavel Paulau
      2. master_events_2.0.log
        2.46 MB
        Pavel Paulau
      3. zoom-reb-large-2.loop_2.0.1-140-rel-enterprise_2.0.0-1976-rel-enterprise_reb_Feb-05-2013_12-36-13.pdf
        1.10 MB
        Ronnie Sun
      1. progress_derivative_2.0.1.png
        52 kB
      2. progress_derivative_2.0.png
        39 kB
      3. progress.png
        49 kB
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        ronnie Ronnie Sun (Inactive) added a comment -

        kv case is much clearer:

        https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdHRRYnlGaTIyQ0VWa3NWTlVqeDZwRFE#gid=8

        We should have algorithm changes in 2.0.1 which causes reb to hang during compaction.

        Please refer to the attached graph (search for 'fragmentation'):

        Compaction kick in both builds, while it did not impact 2.0 at all.

        Show
        ronnie Ronnie Sun (Inactive) added a comment - kv case is much clearer: https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdHRRYnlGaTIyQ0VWa3NWTlVqeDZwRFE#gid=8 We should have algorithm changes in 2.0.1 which causes reb to hang during compaction. Please refer to the attached graph (search for 'fragmentation'): Compaction kick in both builds, while it did not impact 2.0 at all.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -
        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - We've merged a fix: http://review.couchbase.org/#/c/24397/
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Verified for both KV and views cases.

        Show
        pavelpaulau Pavel Paulau added a comment - Verified for both KV and views cases.

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes