Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28847

[FTS] Memory usage during index build overshoots RAM Quota

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.5.0
    • Fix Version/s: 6.0.0
    • Component/s: fts
    • Triage:
      Triaged
    • Is this a Regression?:
      Unknown
    • Sprint:
      FTS Sprint Jul-28-2018, FTS Sprint Aug-11-2018 (Beta)

      Description

      • Build: 2252
      • Indexing YELP Review dataset, 
      • 1 KV node, 2 FTS nodes
      • FTS RAM Quota 1024 (1GB)

      During indexing, stats on the UI shown that RAM usage on a FTS node went up to around 4GB.

      – This is a placeholder bug as of now, I will provide dataset and capture stats, logs as soon as I get access to the FTS perf cluster for another round of experiment.

        Attachments

        1. 1GB_t1t2_rss.png
          1GB_t1t2_rss.png
          44 kB
        2. 1GB_t1t2.png
          1GB_t1t2.png
          29 kB
        3. 2G_t1t2_rss.png
          2G_t1t2_rss.png
          39 kB
        4. 2GB_t1t2.png
          2GB_t1t2.png
          31 kB
        5. 8GB_t1t2_rss.png
          8GB_t1t2_rss.png
          37 kB
        6. 8GB_t1t2.png
          8GB_t1t2.png
          33 kB
        7. 8GB_t1t2.png
          8GB_t1t2.png
          33 kB
        8. mem3.prof
          63 kB
        9. Screen Shot 2018-03-26 at 5.55.02 PM.png
          Screen Shot 2018-03-26 at 5.55.02 PM.png
          420 kB

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            abhinav Abhinav Dangeti added a comment -

            In my local tests, I've found the following settings for the application ratios to show better behavior:

            appRatio: 100%
            indexRatio: 65%
            queryRatio: 85% (need to enable query throttling first, with: http://review.couchbase.org/#/c/97358/)

            Here's the change: http://review.couchbase.org/#/c/97335/9 

            Alex Gyryk Here's a new toybuild for you: http://server.jenkins.couchbase.com/job/toy-unix/3000/artifact/couchbase-server-enterprise-6.0.0-10004-centos7.x86_64.rpm

            Show
            abhinav Abhinav Dangeti added a comment - In my local tests, I've found the following settings for the application ratios to show better behavior: appRatio: 100 % indexRatio: 65 % queryRatio: 85 % (need to enable query throttling first, with: http: //review.couchbase.org/#/c/97358/) Here's the change: http://review.couchbase.org/#/c/97335/9   Alex Gyryk Here's a new toybuild for you:  http://server.jenkins.couchbase.com/job/toy-unix/3000/artifact/couchbase-server-enterprise-6.0.0-10004-centos7.x86_64.rpm
            Hide
            oleksandr.gyryk Alex Gyryk added a comment -

            Index build time, sec

            (40M docs, 40GB on text data indexed):

              Toy build 1 Toy build 2
            1GB quota 2144 2130
            2GB quota 1759 1599
            8GB quota 1100 1200
            150GB quota 1107 1286

            Memory consumption:

            1GB quota:

            2GB quota:

             

             

            8GB quota:

             

             

             

            All reports:

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_80f9&label=1GB-toy2&snapshot=fts_600-10002_build_index_f58c&label=1GB-toy1
            ] [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_aba8&label=2GB-toy2&snapshot=fts_600-10002_build_index_4149&label=2GB-toy1
            ] [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_d00c&label=8GB-toy2&snapshot=fts_600-10002_build_index_a85a&label=8GB-toy1
            ] http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_74af&label=150GB-toy2&snapshot=fts_600-10002_build_index_bff9&label=150GB-toy1

             

            Show
            oleksandr.gyryk Alex Gyryk added a comment - Index build time, sec (40M docs, 40GB on text data indexed):   Toy build 1 Toy build 2 1GB quota 2144 2130 2GB quota 1759 1599 8GB quota 1100 1200 150GB quota 1107 1286 Memory consumption: 1GB quota: 2GB quota:     8GB quota:       All reports: [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_80f9&label=1GB-toy2&snapshot=fts_600-10002_build_index_f58c&label=1GB-toy1 ] [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_aba8&label=2GB-toy2&snapshot=fts_600-10002_build_index_4149&label=2GB-toy1 ] [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_d00c&label=8GB-toy2&snapshot=fts_600-10002_build_index_a85a&label=8GB-toy1 ] http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-10006_build_index_74af&label=150GB-toy2&snapshot=fts_600-10002_build_index_bff9&label=150GB-toy1  
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.5.0-1167 contains cbft commit 225a19e with commit message:
            MB-28847: Memory consumption by the cbft process

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1167 contains cbft commit 225a19e with commit message: MB-28847 : Memory consumption by the cbft process
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.0.0-1449 contains cbft commit 225a19e with commit message:
            MB-28847: Memory consumption by the cbft process

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.0-1449 contains cbft commit 225a19e with commit message: MB-28847 : Memory consumption by the cbft process
            Hide
            abhinav Abhinav Dangeti added a comment -

            Marking as resolved for now, as I don't have any other immediate fixes I'll be making towards this. 

            Show
            abhinav Abhinav Dangeti added a comment - Marking as resolved for now, as I don't have any other immediate fixes I'll be making towards this. 

              People

              • Assignee:
                abhinav Abhinav Dangeti
                Reporter:
                tai.tran Tai Tran
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Gerrit Reviews

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.