Uploaded image for project: 'Couchbase Lite'
  1. Couchbase Lite
  2. CBL-536

Long delay starting replicator, for big databases

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.6.1
    • 2.7.0
    • LiteCore
    • Security Level: Public
    • None
    • iOS Objective-C Couchbase Lite 2.6.0 Community

    Description

      A developer reports long delays, up to several minutes, when a replicator starts before it transfers any documents. During this time there is a large amount of disk I/O. This is a regression in 2.6.

      "I have ... a dev app that has ~1 million documents and is 1.3GB and a staging app that has ~5 million documents and is 4.3GB. The dev app reads about 1GB before pulling documents, whereas the staging app reads 40GB before pulling documents. Reading 40GB takes about 10 minutes.

      A screenshot from the Instruments app attached to his post shows the I/O is caused by c4enum_next, called by litecore::repl::Replicator::_start. There's a missing stack frame in between, which must be Replicator::_findExistingConflicts.

      So this is caused by the scan for pre-existing conflicted documents. Unfortunately there's no index to assist the query C4DocumentEnumerator runs, so it's doing a linear scan of the entire table.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jens Jens Alfke added a comment - - edited

            I first thought this could be fixed by adding an index to the SQLite database:

            CREATE INDEX on kv_default (flags & 2) WHERE (flags & 2) != 0

            but unfortunately it can't be, because for some reason the RecordEnumerator class doesn't test the 'conflicted' flag in its SQLite query — it just scans all the docs and skips the ones that aren't conflicted. (I don't know why. This is code I wrote, so it's my fault, but it's 2+ years old.)

            So the fix will also require updating the RecordEnumerator class to support iterating only conflicted docs (by adding a WHERE clause to the query.)

            jens Jens Alfke added a comment - - edited I first thought this could be fixed by adding an index to the SQLite database: CREATE INDEX on kv_default (flags & 2) WHERE (flags & 2) != 0 but unfortunately it can't be, because for some reason the RecordEnumerator class doesn't test the 'conflicted' flag in its SQLite query — it just scans all the docs and skips the ones that aren't conflicted. (I don't know why. This is code I wrote, so it's my fault, but it's 2+ years old.) So the fix will also require updating the RecordEnumerator class to support iterating only conflicted docs (by adding a WHERE clause to the query.)
            jens Jens Alfke added a comment - I have a fix.

            People

              jens Jens Alfke
              jens Jens Alfke
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty