Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31039

Index disk size is high during initial index build

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Epic Link:
    • Is this a Regression?:
      Yes

      Description

      Build 5.1.2-6022
      Coming from https://issues.couchbase.com/browse/MB-30855?focusedCommentId=289810&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-289810

      Observed that while doing initial index build, index disk size goes high and then settles down after some time.
      For a test of 500M docs, single index build-
      5.1.1-5723 takes maximum of 40GB space, whereas 5.1.2-6022(and vulcan, alice) takes ~100GB space at peak.
      Also observed that higher percentage of fragmentation while initial index build.

      Here is cbmonitor graph- http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=nyx_511-5723_build_secondaryindex_5233&snapshot=nyx_512-6022_build_secondaryindex_8740#ac9466ec456c2a28c48d40e1c4725c3e

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            sundar Sundar Sridharan added a comment -

            This is the same observation for MB-30855.

            With back index optimization in place, indexer is able to load much faster than before, so the plasma log cleaner is unable to keep pace.

            By itself this should not be a bug, since there are hard stops placed at 70% fragmentation.

            Show
            sundar Sundar Sridharan added a comment - This is the same observation for MB-30855 . With back index optimization in place, indexer is able to load much faster than before, so the plasma log cleaner is unable to keep pace. By itself this should not be a bug, since there are hard stops placed at 70% fragmentation.
            Hide
            sundar Sundar Sridharan added a comment -

            duplicate of MB-30855 and not a bug since the disk space is already under check.

            Show
            sundar Sundar Sridharan added a comment - duplicate of MB-30855 and not a bug since the disk space is already under check.
            Hide
            mahesh.mandhare Mahesh Mandhare added a comment -

            The issue is using more memory than spock while initial index build, although it gets down once log cleaner catches up.
            There can be a situation that customer may not have enough space for the disk usage spike we are having while initial index build, it is around 2.5x disk usage.

            Show
            mahesh.mandhare Mahesh Mandhare added a comment - The issue is using more memory than spock while initial index build, although it gets down once log cleaner catches up. There can be a situation that customer may not have enough space for the disk usage spike we are having while initial index build, it is around 2.5x disk usage.
            Hide
            tai.tran Tai Tran added a comment -

            Sundar Sridharan - let's revisit on whether the log cleaning can be configured to be more aggressive.

            Show
            tai.tran Tai Tran added a comment - Sundar Sridharan - let's revisit on whether the log cleaning can be configured to be more aggressive.
            Hide
            sundar Sundar Sridharan added a comment -

            hi Tai Tran, I thought about this too. Fortunately we already have a way of making log cleaner more aggressive with the following:

            indexer.plasma.mainIndex.maxLSSFragmentation
            (default is set to 80%)

            The above setting can be lowered and that will make the log cleaner run more aggressively. However given that this issue only happens during initial load where a DCP optimization speeds up indexing, we may want to think about lowering this threshold. After index load, the fragmentation is kept under check by log cleaner. Making log cleaner more aggressive will likely have an impact on the index load times, so it is a tradeoff that will have to be evaluated.

            My initial thought was the few customers who are worried about temporary spikes in disk space usage can be simply asked to tweak the above setting down?

            Show
            sundar Sundar Sridharan added a comment - hi Tai Tran , I thought about this too. Fortunately we already have a way of making log cleaner more aggressive with the following: indexer.plasma.mainIndex.maxLSSFragmentation (default is set to 80%) The above setting can be lowered and that will make the log cleaner run more aggressively. However given that this issue only happens during initial load where a DCP optimization speeds up indexing, we may want to think about lowering this threshold. After index load, the fragmentation is kept under check by log cleaner. Making log cleaner more aggressive will likely have an impact on the index load times, so it is a tradeoff that will have to be evaluated. My initial thought was the few customers who are worried about temporary spikes in disk space usage can be simply asked to tweak the above setting down?

              People

              • Assignee:
                sundar Sundar Sridharan
                Reporter:
                mahesh.mandhare Mahesh Mandhare
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.