Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29923

[FTS, scorch] high memory consumption by cbft when there are items in persistence queue

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.0.0
    • 5.5.0
    • fts
    • centos 7
    • Untriaged
    • No
    • FTS Sprint Jul-28-2018, FTS Sprint Aug-11-2018 (Beta)

    Description

      My setup:

      node211: data service, 256GB RAM
      node39: FTS service, 10GB RAM, FTS quota  is 1GB 
      node40: FTS service, 10GB RAM, FTS quota is 1GB

      10M documents, wiki dataset
       FTS index size is 14GB, so 7GB per node

       

      Test:

      "wild card" queries and docs mutations (mutation rate is 1K docs/sec)

       

      Behavior:

       

      Phase 1 - indexing. 
      While indexing memory consumption by cbft is up to 3.5GB
      Index is built w/o any issues

       

      Phase 2 - warm-up
      During warm-up phase client runs "wild card" queries only (no kv mutations)
      The memory consumptions remains on 3.5-3.6 GB.
      Consistent query throughput, no issues

       

      Phase 3 - perf test

      Once the amount of items in "Items to persist" starts to grow the cbft memory consumption quickly goes up. With just about 500 items in the queue cbft allocates another 2GB of RAM
      With 1K+ items it takes all available memory (in my setup) and node gets failed-over due to OOM issues 

       

       

      Validating other queries and memory setups...

       

      Logs from one of the runs:

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-enduser-469/172.23.99.211.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-enduser-469/172.23.99.39.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-enduser-469/172.23.99.40.zip

       

       

      Attachments

        1. MB-29923.html
          275 kB
        2. MB-29923.ipynb
          27 kB
        3. Med, Scorch.png
          Med, Scorch.png
          79 kB
        4. node39_perftest_allallocs.svg
          152 kB
        5. out-000.png
          out-000.png
          85 kB
        6. Screen Shot 2018-07-28 at 10.29.05 PM.png
          Screen Shot 2018-07-28 at 10.29.05 PM.png
          90 kB
        7. Screen Shot 2018-07-28 at 10.29.05 PM.png
          Screen Shot 2018-07-28 at 10.29.05 PM.png
          90 kB
        8. Screen Shot 2018-07-28 at 10.48.54 PM.png
          Screen Shot 2018-07-28 at 10.48.54 PM.png
          230 kB
        9. Screen Shot 2018-08-01 at 3.16.09 PM.png
          Screen Shot 2018-08-01 at 3.16.09 PM.png
          161 kB
        10. Wild, Scorch.png
          Wild, Scorch.png
          92 kB
        11. Wild.png
          Wild.png
          71 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Sreekanth Sivasankaran Sreekanth Sivasankaran (Inactive)
              oleksandr.gyryk Alex Gyryk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty