Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-14591

DCP and memory stuck with high DGM.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 4.0.0
    • 4.0.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • 4 node Cluster
      2 nodes KV
      1 node index
      1 node query

    Description

      Steps
      • Create a bucket with 100MB memory quota (A total of 200MB across the cluster)
      • Try creating 20 documents each of size 10MB (A total of 400MB).
      • 8 documents (160MB of data in total) will be created successfully out of 20. The client receives a few tmp_oom then hard out of memory errors.
      • I have attached my test script.

      1d [patrick:~/Code/Python] 17s $ ./set-big.py -m 192.168.78.104  -n 20 -s 10485760
      test_big_0
      test_big_1
      test_big_2
      test_big_3
      test_big_4
      Unexpected error: <class 'couchbase.exceptions.TemporaryFailError'>
      test_big_5
      test_big_6
      Unexpected error: <class 'couchbase.exceptions.TemporaryFailError'>
      test_big_7
      test_big_8
      Unexpected error: <class 'couchbase.exceptions.TemporaryFailError'>
      test_big_9
      test_big_10
      Unexpected error: <class 'couchbase.exceptions.TemporaryFailError'>
      test_big_11
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_12
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_13
      test_big_14
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_15
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_16
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_17
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_18
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      test_big_19
      Unexpected error: <class 'couchbase.exceptions.NoMemoryError'>
      9 Items created successfully
      

      Questions

      I have attached a screenshots highlighting these questions:

      • Where is all the memory being used when we have 0 resident ratio?
      • Why do active and replica items do not match? (See the green circles in the screenshot above).
      • Why does DCP keep getting backing offs when there are no items in the dcp queue?
      Expectations

      I have tested this with both full and value ejection mode behaviour is the same.

      • In value ejection mode hard_oom should only be produce once metadata has filled memory.
      • In full ejection mode hard_oom should never be produce.
      Regression

      I have also tested this on 2.5.2 and it worked as a expected. I was able to store 18 docs with replicas. 2 docs failed with tmp_oom which is expect

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              chiyoung Chiyoung Seo (Inactive)
              pvarley Patrick Varley (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty