Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36804

[BP to 6.0.4] - couchstore node flushing doesn't respect node quota size

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 3.1.6, 4.1.2, 4.5.1, 4.6.5, 5.0.1, 5.1.3, 6.0.2, 6.0.3, 5.5.5
    • 6.0.4
    • storage-engine
    • Triaged
    • Yes

    Description

      Backport of MB-36424 to 6.0.4

      Summary

      When couchstore writes out modified node elements during document saves, it it supposed to limit the number of bytes written in a single node to a _chunk_threshold - by default 1279 bytes. If the node is larger than the limit it should be split into multiple sibling nodes.

      However, this limit is not respected, resulting in overly-large nodes being written out. In the case of the by-seqno B-Tree (which always writes values to the rightmost leaf as seqnos are increasing), it results in all leaf elements residing in a single leaf node. Moreover, this means that adding another element to the B-Tree effectively re-writes the entire tree, resulting in massive Write Amplification.

      Steps to Reproduce

      1. Start 2-node cluster run (single replica):

        ./cluster_run --nodes=2
        

      2. Start a SyncWrite workload; single threaded client updating the same 100,000 items (each key 10 times) with level=persistMajority:

        ./engines/ep/management/sync_repl.py localhost:12000 Administrator asdasd default loop_bulk_setD key value 100000 1000000 3
        

      3. Observe the op/s and Write Amplification

      Expected Results

      • Op/s should be broadly constant (given both key and seqno B-Tree should have a constant number of items in them).
      • Write amplification should also be broadly constant.

      Actual Results

      • The op/s quickly drops from a peak of ~600 down to 150:
      • The Write Amplification increase (corresponding with the top in op/s) from 5.6x, up to 743x .
      • Both op/s and Write Amplification temporarily recover when compaction occurs, but same pattern is observed over time:

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-6.0.4-3020 contains couchstore commit 143858d with commit message:
            MB-36804: Ensure flush_mr_partial() obeys node size quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.4-3020 contains couchstore commit 143858d with commit message: MB-36804 : Ensure flush_mr_partial() obeys node size quota

            Build couchbase-server-6.5.1-6127 contains couchstore commit 143858d with commit message:
            MB-36804: Ensure flush_mr_partial() obeys node size quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6127 contains couchstore commit 143858d with commit message: MB-36804 : Ensure flush_mr_partial() obeys node size quota

            Build couchbase-server-1006.5.1-1065 contains couchstore commit 143858d with commit message:
            MB-36804: Ensure flush_mr_partial() obeys node size quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-1006.5.1-1065 contains couchstore commit 143858d with commit message: MB-36804 : Ensure flush_mr_partial() obeys node size quota

            Build couchbase-server-1006.5.1-1125 contains couchstore commit 143858d with commit message:
            MB-36804: Ensure flush_mr_partial() obeys node size quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-1006.5.1-1125 contains couchstore commit 143858d with commit message: MB-36804 : Ensure flush_mr_partial() obeys node size quota

            Build couchbase-server-7.0.0-2304 contains couchstore commit 143858d with commit message:
            MB-36804: Ensure flush_mr_partial() obeys node size quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2304 contains couchstore commit 143858d with commit message: MB-36804 : Ensure flush_mr_partial() obeys node size quota

            People

              ritesh.agarwal Ritesh Agarwal
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty