Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-17884

speed up docloader and parallelize index definition and build

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Duplicate
    • Major
    • 5.0.0
    • 3.1.3, 4.0.0, 4.1.0, 4.5.0
    • tools
    • Security Level: Public
    • None

    Description

      Docloader has always been known to be slow. The initial approach was just to make it asynchronous, but there are requests for making DB load faster for demos.

      The following is from a test on Mac OS X. Windows and Linux are likely worse because of MB-6232.

      beer-sample…
      Elapsed time: 18.2s
      Load time: 15s
      View definition time: ~1s
      View build time: ~1s

      travel-sample…
      Elapsed time: ~2m
      Unzip time: ~10s
      Load time: ~45s
      Index definition time: ~48s
      Time showing building indexes: ~17s

      It's worth mentioning that the console shows only about 2k ops/s during the data load. It's hard to go that slow usually but the way that tool is built is different.

      See also MB-17879.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. MB-20595 Move cbdocloader to golang and improve performance

          • Major - Major loss of function.
          • Closed
          is triggered by

          Bug - A problem which impairs or prevents the functions of the product. MB-6232 ep-engine needs 1.5 minutes to create 1k vbuckets. Seems too slow (faster with barrier=0)

          • Critical - Crashes, loss of data, severe memory leak.
          • Reopened
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-18933 Improved loading of json documents with cbimport/cbexport

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              anil Anil Kumar (Inactive)
              ingenthr Matt Ingenthron
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty