Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31503

Observing high rebalance time up to 10hrs on 4 node cluster with 2 Billion items and 15k continuous ops

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 6.5.0
    • 6.5.0
    • couchbase-bucket
    • Build : 6.0.0-1529
    • Untriaged
    • Unknown

    Description

      Observing high rebalance time up-to 10hrs  when multiple rebalances (in, out , swap) are made on same 4 node cluster with continuous ops running on the cluster 

      It looks like rebalancing goes on hung status when data compaction kicks in  . Please take a look into this issue 

       

      Data set up :

      1. Initial data load - 2 Billion docs
      2. Continuous operatons - 15 k, 5% Update , 5% Create

      Rebalancing steps :

      1. Step  - Rebalance-In (4-3)
      2. Step  - Rebalance-Out (3-4)
      3. Step - Rebalance Swap ( 4 -4 )

       

       
      Multi Job - 
      http://perf.jenkins.couchbase.com/job/Titan-reb_Multitests/45/
       
      Individual Jobs (unaffected) - 
      http://perf.jenkins.couchbase.com/job/Titan-reb_Multitests/42/
       
      http://perf.jenkins.couchbase.com/job/Titan-reb_Multitests/37/
       
       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sharath.sulochana Sharath Sulochana (Inactive)
              sharath.sulochana Sharath Sulochana (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty