Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19320

300M initial build time is slower in watson than in sherlock/4.1.0

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.5.0
    • 5.0.0
    • forestdb
    • 300M Inital index build time test.
    • Untriaged
    • Unknown

    Description

      Still collecting data.

      Test 4.0.0-4051 4.1.0-5005 4.1.1-5914 4.5.0-2151
      Initial Index Build time 300M (min) 488 372 DNF 633

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-21542 Initial and incremental indexing in Watson is slower than in Sherlock (FDB)

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-19173 Initial and incremental build time regression for 5 indexes (fdb)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-19629 50M Incremental Index build time is (much) slower in Watson than sherlock

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-16046 Regression in 2i initial index build time with 300M items.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-19600 Improve the index build time with append-only writes during initial index build even when circular-writes is the write mode for the index

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            View 8 older comments
            qi Qi Zhu (Inactive) added a comment -

            Updates:

            I just synced up with Cihan on this and Cihan has synced up with Chiyoung/John on this.

            Summary of the discussion:
            This is a tradeoff between disk space and index build time for large index files. it only happens with 5 indexes or 300M items. Considering Watson timeframe, this becomes too late to get fixed. We also have a workload to let CB admin to avoid this issue, which is to do append only writes during initial index build time.
            Chiyoung is planning on multiple improvement on ForestDB in Spock (multiple writher support) which will improve this.

            Based on the above discussion and comments from Cihan in MB19173, I'm moving this bug out of Watson.

            Please let us know if you have any comments/concerns.

            Thanks,
            Qi

            qi Qi Zhu (Inactive) added a comment - Updates: I just synced up with Cihan on this and Cihan has synced up with Chiyoung/John on this. Summary of the discussion: This is a tradeoff between disk space and index build time for large index files. it only happens with 5 indexes or 300M items. Considering Watson timeframe, this becomes too late to get fixed. We also have a workload to let CB admin to avoid this issue, which is to do append only writes during initial index build time. Chiyoung is planning on multiple improvement on ForestDB in Spock (multiple writher support) which will improve this. Based on the above discussion and comments from Cihan in MB19173, I'm moving this bug out of Watson. Please let us know if you have any comments/concerns. Thanks, Qi
            qi Qi Zhu (Inactive) added a comment -

            Assign to Chiyoung. Moved it to Spock.

            Thanks,
            Qi

            qi Qi Zhu (Inactive) added a comment - Assign to Chiyoung. Moved it to Spock. Thanks, Qi
            pavelpaulau Pavel Paulau (Inactive) added a comment -

            Hi Mahesh,

            Please review these results after finishing your work on 4.5.1 benchmarking.

            pavelpaulau Pavel Paulau (Inactive) added a comment - Hi Mahesh, Please review these results after finishing your work on 4.5.1 benchmarking.
            qi Qi Zhu (Inactive) added a comment -

            Thanks, Pavel.

            Mahesh Mandharehere are 2 related tickets/tests:
            https://issues.couchbase.com/browse/MB-19173
            https://issues.couchbase.com/browse/MB-19629

            It will be great to see the comparison before/after.

            Thanks,
            Qi

            qi Qi Zhu (Inactive) added a comment - Thanks, Pavel. Mahesh Mandhare here are 2 related tickets/tests: https://issues.couchbase.com/browse/MB-19173 https://issues.couchbase.com/browse/MB-19629 It will be great to see the comparison before/after. Thanks, Qi
            mahesh.mandhare Mahesh Mandhare (Inactive) added a comment -

            Had offline discussion with Pavel and will review these results after FDB changes.

            mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Had offline discussion with Pavel and will review these results after FDB changes.

            People

              mahesh.mandhare Mahesh Mandhare (Inactive)
              wayne Wayne Siu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty