Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5387

low drain rate in large cluster dgm test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • 1.8.1
    • 1.8.1
    • couchbase-bucket
    • Security Level: Public
    • None
    • centos 5.4 64 bit

    Description

      Install couchbase server 1.8.1-840 on 10 centos 5.x servers 64bit
      Create a 10 nodes cluster with 6.5 GB default bucket.
      Load 30 million items with 256 bytes/item and 10 million items with 512 bytes/item
      Cluster goes in to dgm with 78.8 % resident ratio
      Mutate 40 million items to create cache miss and disk fetch. Maintain the load around 15K to 17K ops.
      Add a node (54) to cluster and rebalance
      Add another node (55) to cluster and rebalance
      Remove node 54 and failover node 55. Then rebalance.
      In all cases above, the drain rate is very slow (around 150 to 300 items per second)
      View drain rate at each node, drain rate is zero at some nodes
      Disk write queue steady increase

      Attachments

        1. 10.3.2.54-8091-diag.txt.gz
          2.88 MB
        2. 10.3.2.55-8091-diag.txt.gz
          1.55 MB
        3. 10.3.2.57-8091-diag.txt.gz
          13.28 MB
        4. 10.3.2.58-8091-diag.txt.gz
          11.60 MB
        5. 10.3.2.59-8091-diag.txt.gz
          11.60 MB
        6. 10.3.2.60-8091-diag.txt.gz
          11.65 MB
        7. 10.3.2.61-8091-diag.txt.gz
          11.51 MB
        8. 10.3.2.62-8091-diag.txt.gz
          11.58 MB
        9. 10.3.2.64-8091-diag.txt.gz
          11.57 MB
        10. 10.3.2.65-8091-diag.txt.gz
          11.61 MB
        11. 10.3.2.66-8091-diag.txt.gz
          11.62 MB
        12. 10.3.2.67-8091-diag.txt.gz
          11.61 MB
        13. Screen Shot 2012-05-28 at 12.58.21 PM.png
          Screen Shot 2012-05-28 at 12.58.21 PM.png
          77 kB
        14. Screen Shot 2012-05-28 at 7.11.43 PM.png
          Screen Shot 2012-05-28 at 7.11.43 PM.png
          91 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            chiyoung Chiyoung Seo (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty