Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11488

KV+XDCR System test: mem_used on each node reached to the bucket quota and rebalance hangs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 3.0-Beta, 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • CentOS 6.x 8*8 clusters. Each node : 15GB RAM, 450Gb HDD
    • Untriaged
    • Unknown
    • June 30 - July 18

    Description

      Continuation on MB-11448 [This system test was modified to maintain 25% active resident item ratio compared to the previous run which was <1% ]

      Build
      --------
      3.0.0-840(xdcr on upr, internal replication on upr)

      Clusters
      -----------
      Source : http://172.23.105.44:8091/
      Destination : http://172.23.105.54:8091/
      The clusters are available to investigate.

      Steps
      --------
      1. Load on both clusters till vb_active_resident_items_ratio < 50.
      2. Access phase with 50% gets, 50%deletes runs for 3 hours

      Bucket under discussion: standardbucket (has uni-xdcr)
      Another bucket standardbucket1 also has uni-xdcr but it's memory usage is comparable to that of other nodes. So it may not be due to XDCR.

      Bucket capacity:5GB but ep_value_size is >10gb while the node is at 0% dgm.
      Rest of the nodes are at 75% dgm with 3.8GB.

      Cluster C1: http://172.23.105.44:8091/
      C2 : http://172.23.105.54:8091/

      See attached screenshots and logs. Marking this as a test blocker because rebalance in this scenario will hang.

      Attachments

        1. Screen Shot 2014-06-19 at 5.36.42 PM.png
          298 kB
          Aruna Piravi
        2. Screen Shot 2014-06-19 at 5.36.54 PM.png
          288 kB
          Aruna Piravi
        3. Screen Shot 2014-06-19 at 5.37.11 PM.png
          426 kB
          Aruna Piravi
        4. Screen Shot 2014-06-20 at 3.41.51 PM.png
          165 kB
          Abhi Dangeti
        5. Screen Shot 2014-06-23 at 6.49.12 PM.png
          129 kB
          Aruna Piravi
        6. Screen Shot 2014-06-23 at 6.49.25 PM.png
          239 kB
          Aruna Piravi
        7. Screen Shot 2014-06-25 at 11.35.27 AM.png
          178 kB
          Abhi Dangeti
        8. Screen Shot 2014-06-29 at 12.01.59 AM.png
          326 kB
          Andrei Baranouski

        Issue Links

          For Gerrit Dashboard: MB-11488
          # Subject Branch Project Status CR V

          Activity

            People

              chiyoung Chiyoung Seo (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty