Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31503

Observing high rebalance time up to 10hrs on 4 node cluster with 2 Billion items and 15k continuous ops

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 6.5.0
    • 6.5.0
    • couchbase-bucket
    • Build : 6.0.0-1529
    • Untriaged
    • Unknown

    Description

      Observing high rebalance time up-to 10hrs  when multiple rebalances (in, out , swap) are made on same 4 node cluster with continuous ops running on the cluster 

      It looks like rebalancing goes on hung status when data compaction kicks in  . Please take a look into this issue 

       

      Data set up :

      1. Initial data load - 2 Billion docs
      2. Continuous operatons - 15 k, 5% Update , 5% Create

      Rebalancing steps :

      1. Step  - Rebalance-In (4-3)
      2. Step  - Rebalance-Out (3-4)
      3. Step - Rebalance Swap ( 4 -4 )

       

       
      Multi Job - 
      http://perf.jenkins.couchbase.com/job/Titan-reb_Multitests/45/
       
      Individual Jobs (unaffected) - 
      http://perf.jenkins.couchbase.com/job/Titan-reb_Multitests/42/
       
      http://perf.jenkins.couchbase.com/job/Titan-reb_Multitests/37/
       
       

      Attachments

        1. Screen Shot 2018-10-05 at 15.15.29.png
          46 kB
          Ben Huddleston
        2. Screen Shot 2018-10-05 at 15.17.44.png
          55 kB
          Ben Huddleston
        3. Screen Shot 2018-10-05 at 15.45.13.png
          59 kB
          Ben Huddleston
        4. Screen Shot 2018-10-05 at 16.05.56.png
          179 kB
          Ben Huddleston
        5. Screen Shot 2018-10-05 at 16.10.05.png
          392 kB
          Ben Huddleston
        6. Screen Shot 2018-10-08 at 10.00.35.png
          123 kB
          Ben Huddleston
        7. Screen Shot 2018-10-08 at 10.04.00.png
          87 kB
          Ben Huddleston
        8. Screen Shot 2018-10-08 at 10.06.11.png
          179 kB
          Ben Huddleston
        9. Screen Shot 2018-10-10 at 17.10.43.png
          223 kB
          Ben Huddleston
        10. Screen Shot 2018-10-11 at 16.49.49.png
          318 kB
          Ben Huddleston
        11. Screen Shot 2018-10-11 at 16.49.59.png
          495 kB
          Ben Huddleston
        12. Screen Shot 2018-10-22 at 15.26.16.png
          221 kB
          Ben Huddleston

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sharath.sulochana Sharath Sulochana (Inactive)
              sharath.sulochana Sharath Sulochana (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty