Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6981

Beta 2.0 - high fragmentation levels on GAT operations

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0-beta
    • Fix Version/s: bug-backlog
    • Security Level: Public
    • Labels:
      None

      Description

      From here: http://www.couchbase.com/forums/thread/beta-2-0-high-fragmentation-levels-gat-operations

      Hi everyone,
      In our application we write fairly large K/V pairs - about 60Kbyte each. These are image chips. We set a fairly long TTL on these - 1 week.
      When we need a image chip we request it from Couchbase via a GAT operation. If it exists we re-set the TTL to 1 week with the goal to keep those chips access alive in the cache, and let the stale content expire.
      The problem we are seeing is that a GAT operation does not simply update the TTL in the metadata but causes the entire K/V (with new TTL) to be appended to the end of the disk file. This causes fragmentation to grow quite quickly of the document store, requiring lots of compactions to keep it in control.
      So my question is why does a TTL update (metadata) even require this to occur and is there a way to tune around this?
      Thanks

      We should be able to update just the meta-data for GAT, instead of rewriting the whole doc.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        chiyoung Chiyoung Seo added a comment -

        To resolve this issue, we need an additional API from couchstore, which allows us to update the meta only for a given key.

        If couchstore already provides the update_meta(..), please assign it back to me.

        Show
        chiyoung Chiyoung Seo added a comment - To resolve this issue, we need an additional API from couchstore, which allows us to update the meta only for a given key. If couchstore already provides the update_meta(..), please assign it back to me.
        Hide
        damien damien added a comment -

        Aaron, what will it take for an API to update metadata only? If it's already there, let Chiyoung know where to find the API info and reassign to him.

        Show
        damien damien added a comment - Aaron, what will it take for an API to update metadata only? If it's already there, let Chiyoung know where to find the API info and reassign to him.
        Hide
        aaron Aaron Miller (Inactive) added a comment -

        Huh. I'd actually done a patch to add this feature to couchstore once before, for this exact use case. For some reason it was decided that this wasn't needed, and I guess the couchstore patch was never merged. I can do it again pretty easily.

        Show
        aaron Aaron Miller (Inactive) added a comment - Huh. I'd actually done a patch to add this feature to couchstore once before, for this exact use case. For some reason it was decided that this wasn't needed, and I guess the couchstore patch was never merged. I can do it again pretty easily.
        Hide
        aaron Aaron Miller (Inactive) added a comment -

        Here you go: http://review.couchbase.org/#/c/21859/

        If COUCHSTORE_DOCINFO_UPDATE is passed in options and the corresponding passed document pointer is NULL, the DocInfo will be updated as-is, (preserving the current body pointer), and will not rewrite the body.

        Show
        aaron Aaron Miller (Inactive) added a comment - Here you go: http://review.couchbase.org/#/c/21859/ If COUCHSTORE_DOCINFO_UPDATE is passed in options and the corresponding passed document pointer is NULL, the DocInfo will be updated as-is, (preserving the current body pointer), and will not rewrite the body.
        Hide
        chiyoung Chiyoung Seo added a comment - - edited

        We discussed this issue today and decided to move it to 2.0.2 or later. This requires some changes in the ep-engine flusher, KVStore, and TAP replication message format to indicate the meta data update only. It also affects the 2.0 backup tools.

        Show
        chiyoung Chiyoung Seo added a comment - - edited We discussed this issue today and decided to move it to 2.0.2 or later. This requires some changes in the ep-engine flusher, KVStore, and TAP replication message format to indicate the meta data update only. It also affects the 2.0 backup tools.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Mike/Chiyoung,

        is this a must-have for 3.0? pls update this bug.

        Show
        maria Maria McDuff (Inactive) added a comment - Mike/Chiyoung, is this a must-have for 3.0? pls update this bug.
        Hide
        mikew Mike Wiederhold added a comment -

        I don't think it is a must have, but it would really be nice to get this fixed since it will reduce the need for compaction in certain use cases.

        Show
        mikew Mike Wiederhold added a comment - I don't think it is a must have, but it would really be nice to get this fixed since it will reduce the need for compaction in certain use cases.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Post 3.0

        Show
        maria Maria McDuff (Inactive) added a comment - Post 3.0
        Hide
        mikew Mike Wiederhold added a comment -

        This might be good to get addressed in the forestdb kvstore.

        Show
        mikew Mike Wiederhold added a comment - This might be good to get addressed in the forestdb kvstore.

          People

          • Assignee:
            sriram Sriram Ganesan
            Reporter:
            damien damien
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Gerrit Reviews

              There are no open Gerrit changes