Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11609

KV+XDCR System test: Uneven active resident ratio(0.2% to 25%) on cluster nodes, suddenly drops to 0%

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • centOS 64 bit each node - 5 cores 15GB
    • Untriaged
    • Unknown
    • June 30 - July 18

    Description

      Scenario
      -------------
      1. Load on both clusters till vb_active_resident_items_ratio < 50 on standardbucket, <70 on standardbucket1, 20M on sasl.
      2. Access phase with 100% gets runs for 3 hours with 50% gets and 50% deletes
      3. Rebalance-out 1 node at cluster1
      4. Rebalance-in 1 node at cluster1
      5. Failover and remove node at cluster1
      6. Failover and add-back node at cluster1
      7. Rebalance-out 1 node at cluster2
      8. Rebalance-in 1 node at cluster2
      9. Failover and remove node at cluster2
      10. Failover and add-back node at cluster2
      11. Soft restart all nodes in cluster1 one by one
      12. Soft restart all nodes in cluster2 one by one

      During phase 11 it was seen that cluster 2 had uneven resident ratio(one node .55 had 0.2% while others ranging from 8-25%) . During phase 12, active resident ratio suddenly dropped to 0 on all nodes. Test log shows cluster warmup was issued, however, no such activity is apparent from GUI logs. KIndly check if warmup resulted in this state.

      Please see from the screenshot that no nodes have reached 5GB memory usage (allotted bucket capacity) yet active resident ratio for all nodes is 0. I've previously loaded upto 150M docs(same size) onto 5GB buckets without hitting this issue. Looks like we have some memory bloating happening.

      Attaching cbcollect info. Live cluster : http://172.23.105.54:8091/index.html

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            chiyoung Chiyoung Seo (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty