Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6726

Rebalance is slow when indexing/compaction and query load are going on in parallel


    • Sprint:
      PCI Team - Sprint 1, PCI Team - Sprint 5


      Cluster information:

      • 8 centos 6.2 64bit server with 4 cores CPU
      • Each server has 32 GB RAM and 400 GB SSD disk.
      • SSD disk format ext4 on /data
      • Each server has its own drive, no disk sharing with other server.
      • Cluster has 2 buckets, default (11GB) and saslbucket (11GB) with consistent view enable. For 2 buckets, we use only 68% total RAM of system.
      • Load 12 million items to saslbucket and 45 million items to default bucket. Each key has size from 512 bytes to 1500 bytes
      • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
      • Create cluster with 4 nodes installed couchbase server 2.0.0-1746

      • Data path /data
      • View path /data
      • Let load running at 6K ops, query 400 to 500 per second
      • Add 4 nodes to cluster and rebalance
      • rebalance hang. Filed bug MB-6706
      • Try rebalance again. Cluster rebalances saslbucket first and took 9 hours to complete rebalance saslbucket
      • default bucket took 32 hours to complete

      Link to collect_info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-slow-constn-enable-20120925-141314.tgz

      Each atop file size from 1.5 GB to 2.5 GB. If atop file needed, I will upload them as request

      1. master_events.log.svg
        246 kB
        Aleksey Kondratenko
      2. master_events.tar.gz
        1.05 MB
        Pavel Paulau
      1. rebalance_progress.png
        37 kB

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.


          No work has yet been logged on this issue.


            • Assignee:
              thuan Thuan Nguyen
              thuan Thuan Nguyen
            • Votes:
              0 Vote for this issue
              14 Start watching this issue


              • Created: