Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55530

Size seqIndex data level based on uncompressed docs size

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Magma issues compactions based on the size ratios of L3:L4 level in seqIndex. L4 stores the actual user data and L3 carries logical deletes for that user data. When this ratio exceeds the configured fragmentation threshold (50%), a compaction is triggered.

      However when computing the size ratio, we compare block uncompressed size of L3 against block compressed size of L4. 

      For highly compressible docs, this can result in size of L3 >>> L4 which actually doesn't make sense as we cannot have more deletes than data. This results in compactions kicking in much earlier than required and increasing write amp. For example, if we load 1GB of data that gets compressed to only 100MB, size of L4 will be 100MB. Now if we overwrite 100MB of docs (10% of docs), L3 will be 100MB, whereas L4 will be (100MB loaded data+10MB of newly written data=110 MB), giving a fragmentation ratio of 90% even though we only overwrote 10% of docs.

      To fix this we need to size L4 also using uncompressed size. 

      This also breaks some of our stats currently. As whenever Magma sees L3>L4, it treats fragmentation as 0, even though it will start doing compactions. 

      This also requires some stats changes in kv-engine/ns-server:

      • db_data_size will now be the uncompressed data size. It is now the compressed size of unfragmented data. If this is a breaking change, then better to add a new db_data_uncompressed_size.
      • a new stat db_file_uncompressed_size will be added.
      • the above two stats will be used for couch_docs_fragmentation computation in ns-server.
      • db_file_size will continue to be the compressed size.

      As part of this we'll also add two stats for history:

      • db_history_file_uncompressed_size
      • db_history_data_uncompressed_size
      • couch_docs_fragmentation will then be =

      (db_file_uncompressed_size-db_history_file_uncompressed_size)-(db_data_uncompressed_size-db_history_data_uncompressed_size)/(db_file_uncompressed_size-db_history_file_uncompressed_size)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ankush.sharma Ankush Sharma
              rohan.suri Rohan Suri
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty