Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31405

[FTS] high disk usage during initial indexing in DGM scenario

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Is this a Regression?:
      No
    • Sprint:
      FTS Sprint Nov-23-2018

      Description

      .. and also weird behavior.

      The test:  Initial indexing fo 40M dataset, single FTS node, 32GB RAM, various quotas.
      First run was with very high FTS quota to measure the max RAM fts would use when initial indexing.
      The experiment shows that on single FTS node, to support its maximum indexing throughput of 40 MB/sec  the fts uses 10GB of RAM

      Having that in mind, I ran few tests with quota lower that 10GB considering that tests as "DGM" kind of scenarios.

      The behavior is: the higher the quota within those 10GB the slower the indexing and the more disk space it uses.

       

      First, the indexing duration:

      Once we get into DGM, the indexing throughput drops 4-5 times. And the higher the quota the slower it gets:

      Quota, GB duration, sec
      >10 GB 1219
      1 4301
      2 4991
      4 4971
      6 6266
      8 5498

       

      Same is with disk size, but even on larger scale:

      Quota, GB max disk usage, GB
      >10 GB 112
      1 103
      2 177
      4 178
      6 423
      8 347

       

      Here is comparison of disk usage pattern with various quota sizes:

      Full comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-1614_build_index_ec7d&label=1%20GB&snapshot=fts_600-1614_build_index_3fc7&label=2%20GB&snapshot=fts_600-1614_build_index_f2fc&label=4%20GB&snapshot=fts_600-1614_build_index_7a37&label=6%20GB&snapshot=fts_600-1614_build_index_82f5&label=%3E10%20GB

       

      Its getting fun when comparing how system behaves with 10GB and 8GB quota assuming that we just 20% less that max required:

      Especially because it uses the same amount of memory

       

      Logs:

       

       

       

       

       

        Attachments

          Issue Links

          For Gerrit Dashboard: MB-31405
          # Subject Branch Project Status CR V

            Activity

            Hide
            oleksandr.gyryk Alex Gyryk added a comment -

            This graph shows nice picture of how indexing speed changes once we go under DGM:

            Show
            oleksandr.gyryk Alex Gyryk added a comment - This graph shows nice picture of how indexing speed changes once we go under DGM:
            Hide
            Sreekanth Sivasankaran Sreekanth Sivasankaran added a comment - - edited

            Thanks Alex Gyryk for these nice graphs. If we could capture the trend of stats like  num_root_memorysegments and num_root_filesegments, that may give more insights into this.
             
            Its difficult to categorically theorise these findings. But few non proven intuitions are,
             
            1. With 10GB RAM/perfect configs seen so far, we are able to tune the performance of three main actor trio in the system introducer/persister/merger. They step together to achieve a better balance between memory, IO and CPU.

            Above point is no longer valid, Steve Yen is right, I mis-understood that with 10GB everything worked fine. 

            Here for indexing alone, we may be able to find a optimum FTS quota (somewhere bw 10 - 14GB) .

            Similarly for querying alone, we may find an optimum FTS quota. May be there are two cases as we need to consider both latency and QPS.

            And these three FTS quotas, can guide us to formulating a reasonable/recommendable FTS quota for a mixed load of indexing and querying, Alex Gyryk any thoughts on this?
             
            2. With lower memory quota, app_herder pressure kicks in very often. This results in the flush of memory segments pretty soon (along with some in memory merging which would have reduced the in memory segment count) to disk. And the in memory merging along with the short memory flush times triggered by app_herder, result in persistence of lesser number of files to disk.  Lesser number of files also helps in faster file based merging to eventually reduce the file count/disk usage in a better way. All this leads to faster unblocking of app_herder/mitigation of the memory pressure situation and the system gets back into the indexing mode quickly.
             
            3. With high memory quota nearing the optimum value 10, as there is more memory - the app_herder interventions are going to less frequent. This results in higher number of in-memory segments even with in memory merging. Also, as and when we have more in-memory segments, the memory merging ought to take more time as it involves merging + persistence of more segment data. All in all, we end up having more segments in memory with higher memory quota which gets flushed to disk upon app_herder pause, and more file segments puts more load on file merging to eventually results in more files on disk.  More files on disk leads to heavier file merging + IO boundedness. All this leads to slower unblocking of app_herder/mitigation of the memory pressure situation and the system unblocks the indexing slowly.
             
            Steve Yen, may have better insights here.. will sync for mad-hatter..
             

            Show
            Sreekanth Sivasankaran Sreekanth Sivasankaran added a comment - - edited Thanks Alex Gyryk for these nice graphs. If we could capture the trend of stats like  num_root_memorysegments and num_root_filesegments, that may give more insights into this.   Its difficult to categorically theorise these findings. But few non proven intuitions are,   1. With 10GB RAM/perfect configs seen so far, we are able to tune the performance of three main actor trio in the system introducer/persister/merger. They step together to achieve a better balance between memory, IO and CPU. Above point is no longer valid, Steve Yen  is right, I mis-understood that with 10GB everything worked fine.  Here for indexing alone, we may be able to find a optimum FTS quota (somewhere bw 10 - 14GB) . Similarly for querying alone, we may find an optimum FTS quota. May be there are two cases as we need to consider both latency and QPS. And these three FTS quotas, can guide us to formulating a reasonable/recommendable FTS quota for a mixed load of indexing and querying, Alex Gyryk  any thoughts on this?   2. With lower memory quota, app_herder pressure kicks in very often. This results in the flush of memory segments pretty soon (along with some in memory merging which would have reduced the in memory segment count) to disk. And the in memory merging along with the short memory flush times triggered by app_herder, result in persistence of lesser number of files to disk.  Lesser number of files also helps in faster file based merging to eventually reduce the file count/disk usage in a better way. All this leads to faster unblocking of app_herder/mitigation of the memory pressure situation and the system gets back into the indexing mode quickly.   3. With high memory quota nearing the optimum value 10, as there is more memory - the app_herder interventions are going to less frequent. This results in higher number of in-memory segments even with in memory merging. Also, as and when we have more in-memory segments, the memory merging ought to take more time as it involves merging + persistence of more segment data. All in all, we end up having more segments in memory with higher memory quota which gets flushed to disk upon app_herder pause, and more file segments puts more load on file merging to eventually results in more files on disk.  More files on disk leads to heavier file merging + IO boundedness. All this leads to slower unblocking of app_herder/mitigation of the memory pressure situation and the system unblocks the indexing slowly.   Steve Yen , may have better insights here.. will sync for mad-hatter..  
            Hide
            steve Steve Yen added a comment -

            Indeed! Alex Gyryk – this is a great ticket – the graphs are terrific!

            > More files on disk leads to heavier file merging + IO boundedness. All this leads to slower unblocking of app_herder/mitigation of the memory pressure situation and the system unblocks the indexing slowly.

            It does seem the graphs are confirming Sreekanth Sivasankaran's theory... they're showing up to 50,000 files on disk.

            Not sure what to do about it, except perhaps try to think about the heuristics a bit more to somehow avoid 50,000 files? And, also retargeting this out of Alice and over to Mad Hatter, so there's more breathing room to think about that.

            I suppose the production workaround advice in Alice is telling folks to either have everything fit into RAM (the >10GB scenario?) or instead provide a lower quota to avoid a huge # of files when initial DGM indexing is important. And, that folks should test their own situations, too!

            Show
            steve Steve Yen added a comment - Indeed! Alex Gyryk – this is a great ticket – the graphs are terrific! > More files on disk leads to heavier file merging + IO boundedness. All this leads to slower unblocking of app_herder/mitigation of the memory pressure situation and the system unblocks the indexing slowly. It does seem the graphs are confirming Sreekanth Sivasankaran 's theory... they're showing up to 50,000 files on disk. Not sure what to do about it, except perhaps try to think about the heuristics a bit more to somehow avoid 50,000 files? And, also retargeting this out of Alice and over to Mad Hatter, so there's more breathing room to think about that. I suppose the production workaround advice in Alice is telling folks to either have everything fit into RAM (the >10GB scenario?) or instead provide a lower quota to avoid a huge # of files when initial DGM indexing is important. And, that folks should test their own situations, too!
            Hide
            steve Steve Yen added a comment -

            btw, I'm reading the ">10GB scenario" (and the super fast red line on the graph) to mean that the quota was set really high so that led the app-herder to basically never kick in during indexing, and so that all the incoming data was basically ingested in-memory.

            Show
            steve Steve Yen added a comment - btw, I'm reading the ">10GB scenario" (and the super fast red line on the graph) to mean that the quota was set really high so that led the app-herder to basically never kick in during indexing, and so that all the incoming data was basically ingested in-memory.
            Hide
            oleksandr.gyryk Alex Gyryk added a comment - - edited

            Actually quota was 16GB in that experiment but since it never went above 11-12GB during the entire indexing process, yes I assume app-herder never kick in.

            Show
            oleksandr.gyryk Alex Gyryk added a comment - - edited Actually quota was 16GB in that experiment but since it never went above 11-12GB during the entire indexing process, yes I assume app-herder never kick in.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.5.0-1637 contains bleve commit a99c919 with commit message:
            Merge pull request #1023 from sreekanth-cb/merger_fix

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1637 contains bleve commit a99c919 with commit message: Merge pull request #1023 from sreekanth-cb/merger_fix
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.5.0-1637 contains bleve commit 2e631fb with commit message:
            MB-31405 - higher disk usage in DGM

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1637 contains bleve commit 2e631fb with commit message: MB-31405 - higher disk usage in DGM
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.0.1-1969 contains bleve commit 06435f1 with commit message:
            MB-31405 - higher disk usage in DGM

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.1-1969 contains bleve commit 06435f1 with commit message: MB-31405 - higher disk usage in DGM

              People

              • Assignee:
                oleksandr.gyryk Alex Gyryk
                Reporter:
                oleksandr.gyryk Alex Gyryk
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.