Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29217

9 node rebalance is 5-8x slower in vulcan

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      http://showfast.sc.couchbase.com/#/timeline/Linux/reb/kv/DGM

      The master_events.log indicates that vbucket spends more time in seqnoWaitingStarted state

       

      If comparing to 5.1 and older builds it seems like more and more vbuckets getting affected:

      Amount of "slow" buckets is random so test results aren't consistent.

      We weren't able to identify the changes in Vulcan which might cause this behavior.

       

      5.1 logs:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-140/172.23.96.105.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-140/172.23.96.100.zip

      5.5 logs:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-188/172.23.96.105.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-188/172.23.96.100.zip

       

       

       

      Attachments

        1. node-109-mem_used.png
          148 kB
          Paolo Cocchi
        2. rebalance_510-5552vs550-2361vs550-2432.png
          110 kB
          Paolo Cocchi
        3. rebalance_550-2432vs550-2709.png
          97 kB
          Paolo Cocchi
        4. rebalance_550-2432vs550-2709-2.png
          116 kB
          Paolo Cocchi
        5. rebalance_550-2734_251vs252.png
          236 kB
          Paolo Cocchi
        6. Screen Shot 2018-04-20 at 2.34.48 PM.png
          63 kB
          Alex Gyryk

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-26791 Latency of GET operations exceeds minutes during swap rebalance (9 nodes, 1TB)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-29682 Rebalance exited with reason {child_died,bad_replicas}

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              paolo.cocchi Paolo Cocchi
              oleksandr.gyryk Alex Gyryk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty