Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7053

Expired items are not excluded from production/dev views until expiry pager runs ( which deletes the items from database permanently )

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0
    • Component/s: None
    • Security Level: Public
    • Environment:

      Description

      Expired items are not excluded from production/dev views. This happens only after the item is "get".

      Steps to reproduce:

      1. Create default bucket and create 1 production view
      curl -X PUT -H 'Content-Type: application/json' 'http://Administrator:asdasd@127.0.0.1:9500/default/_design/d1' -d '{"views":{"v1":{"map":"function(doc,meta)

      {\nemit(meta.id,doc);\n}

      "}}}'

      {"ok":true,"id":"_design/d1"}

      2. Insert 2 items from memcached client with expiry set to 5 seconds

      >>> import mc_bin_client
      >>> mc = mc_bin_client.MemcachedClient(port=12001)
      >>> mc.set("ab", 5, 0, "val")
      (3306888435, 7973028514358, '')
      >>> mc.set("ab1", 5, 0, "val")
      (2896330942, 7997559610975, '')

      3. Query the view with stale=false to build the index
      curl -X GET 'http://127.0.0.1:9500/default/_design/d1/_view/v1?stale=false'
      {"total_rows":2,"rows":[

      {"id":"ab","key":"ab","value":"dmFs"}

      ,

      {"id":"ab1","key":"ab1","value":"dmFs"}

      ]
      }

      4. Query the view after couple of minutes and expired items are still returned
      curl -X GET 'http://127.0.0.1:9500/default/_design/d1/_view/v1?stale=false'
      {"total_rows":2,"rows":[

      {"id":"ab","key":"ab","value":"dmFs"}

      ,

      {"id":"ab1","key":"ab1","value":"dmFs"}

      ]
      }

      If memcached get is used with these items, then these are excluded from the views. Otherwise these are always returned in view results.

      Diagnostic is attached.

      Checked with Mike on this and ep-engine seems to be doing things correctly:

      "This does look like an issue, but not in the ep-engine side. Since ep-engine might take an hour to actually remove an expired item, it should be up to the view engine to filter out any expired items too. The reaon why doing a get will cause the item to disappear from the view results is that ep-engine will actually do the deletion."

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        deepkaran.salooja Deepkaran Salooja created issue -
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Won't work unless the docs are deleted from the vbucket databases.

        See:

        http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them#Debuggingviewengineissuesandreportingthem-section5

        Simply doesn't work for several architectural reasons.
        See the discussion there.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Won't work unless the docs are deleted from the vbucket databases. See: http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them#Debuggingviewengineissuesandreportingthem-section5 Simply doesn't work for several architectural reasons. See the discussion there.
        FilipeManana Filipe Manana (Inactive) made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        farshid Farshid Ghods (Inactive) made changes -
        Resolution Won't Fix [ 2 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Assignee Filipe Manana [ filipemanana ] MC Brown [ mccouch ]
        farshid Farshid Ghods (Inactive) made changes -
        Summary Expired items are not excluded from production/dev views Expired items are not excluded from production/dev views until expiry pager runs ( which deletes the items from database permanently )
        Component/s documentation [ 10012 ]
        Component/s view-engine [ 10060 ]
        farshid Farshid Ghods (Inactive) made changes -
        Labels 2.0-release-notes
        Affects Version/s recent-builds-2.0 [ 10349 ]
        Hide
        dipti Dipti Borkar added a comment -

        At least a client side solution needs to be put in.

        Show
        dipti Dipti Borkar added a comment - At least a client side solution needs to be put in.
        dipti Dipti Borkar made changes -
        Assignee MC Brown [ mccouch ] Rahim Yaseen [ yaseen ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        deepkaran.salooja Deepkaran Salooja added a comment -
        Show
        deepkaran.salooja Deepkaran Salooja added a comment - Another user on the forum facing same issue: http://www.couchbase.com/forums/thread/expired-items-still-appear-document-list
        Hide
        deepkaran.salooja Deepkaran Salooja added a comment -

        Copying full email conversation with Mike on this.

        On Oct 29, 2012, at 11:56 AM, Mike Wiederhold <mike@couchbase.com> wrote:

        Yes, that's what I would expect.

        • Mike

        On Oct 29, 2012, at 11:53 AM, Farshid Ghods wrote:

        okay in that case view index should be updated and stale=false query should pick that up

        On Oct 29, 2012, at 11:46 AM, Mike Wiederhold <mike@couchbase.com> wrote:

        Yes expiration time is updated on disk by ep-engine.

        • Mike

        On Oct 29, 2012, at 11:44 AM, Farshid Ghods wrote:

        Mike,

        does ep-engine update the expiration time on for this key on the disk ?
        if so then view-engine can skip this item when generating the results i guess

        -Farshid

        On Oct 29, 2012, at 11:41 AM, Mike Wiederhold <mike@couchbase.com> wrote:

        This does look like an issue, but not in the ep-engine side. Since ep-engine might take an hour to actually remove an expired item, it should be up to the view engine to filter out any expired items too. The reaon why doing a get will cause the item to disappear from the view results is that ep-engine will actually do the deletion.

        • Mike
        Show
        deepkaran.salooja Deepkaran Salooja added a comment - Copying full email conversation with Mike on this. On Oct 29, 2012, at 11:56 AM, Mike Wiederhold <mike@couchbase.com> wrote: Yes, that's what I would expect. Mike On Oct 29, 2012, at 11:53 AM, Farshid Ghods wrote: okay in that case view index should be updated and stale=false query should pick that up On Oct 29, 2012, at 11:46 AM, Mike Wiederhold <mike@couchbase.com> wrote: Yes expiration time is updated on disk by ep-engine. Mike On Oct 29, 2012, at 11:44 AM, Farshid Ghods wrote: Mike, does ep-engine update the expiration time on for this key on the disk ? if so then view-engine can skip this item when generating the results i guess -Farshid On Oct 29, 2012, at 11:41 AM, Mike Wiederhold <mike@couchbase.com> wrote: This does look like an issue, but not in the ep-engine side. Since ep-engine might take an hour to actually remove an expired item, it should be up to the view engine to filter out any expired items too. The reaon why doing a get will cause the item to disappear from the view results is that ep-engine will actually do the deletion. Mike
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        One thing i am not quite sure about ep-engine behavior is whether the disk is updated before expiry pager runs ?

        Yaseen,
        can we confirm this with ep-engine folks or ideally if there is a spec about how ep-engine handles expirations before/after expiry pager runs

        Show
        farshid Farshid Ghods (Inactive) added a comment - One thing i am not quite sure about ep-engine behavior is whether the disk is updated before expiry pager runs ? Yaseen, can we confirm this with ep-engine folks or ideally if there is a spec about how ep-engine handles expirations before/after expiry pager runs
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Dipti/Yaseen,

        this is a change that can be addressed post 2.0 instead of changing the view-engine or ep-engine at this stage , and i agree that it is confusing for the users so maybe we can simply change how often expiry pager runs if they want faster updates ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - Dipti/Yaseen, this is a change that can be addressed post 2.0 instead of changing the view-engine or ep-engine at this stage , and i agree that it is confusing for the users so maybe we can simply change how often expiry pager runs if they want faster updates ?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Dipti/Yaseen

        Filipe has also explained the options in more details here as why skipping results in the time of reduction or view results wont work. so this bug is basically a duplicate.

        http://www.couchbase.com/issues/browse/MB-6219?focusedCommentId=36489&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-36489

        Show
        farshid Farshid Ghods (Inactive) added a comment - Dipti/Yaseen Filipe has also explained the options in more details here as why skipping results in the time of reduction or view results wont work. so this bug is basically a duplicate. http://www.couchbase.com/issues/browse/MB-6219?focusedCommentId=36489&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-36489
        farshid Farshid Ghods (Inactive) made changes -
        Assignee Rahim Yaseen [ yaseen ] Steve Yen [ steve ]
        farshid Farshid Ghods (Inactive) made changes -
        Assignee Steve Yen [ steve ] Rahim Yaseen [ yaseen ]
        Hide
        steve Steve Yen added a comment -

        options...

        • idea from Damien (might repeat something Filipe had on MB-6219?): track expiration time in secondary index.
        • run expiry pager more often - this scans entire hashtable, locking each hashtable partition at a time. This only reduces the window of the issue, but doesn't fundamentally solve the issue. Ask some customers set their expiry pager more often (e.g., once every 10 minutes for small/medium cluster).
        Show
        steve Steve Yen added a comment - options... idea from Damien (might repeat something Filipe had on MB-6219 ?): track expiration time in secondary index. run expiry pager more often - this scans entire hashtable, locking each hashtable partition at a time. This only reduces the window of the issue, but doesn't fundamentally solve the issue. Ask some customers set their expiry pager more often (e.g., once every 10 minutes for small/medium cluster).
        Hide
        steve Steve Yen added a comment -

        For 2.0, options discussed in bug-scrub...

        Recommend to impacted customers to run expiry pager more often – e.g., once every 10 minutes for small/medium clusters. This can help mitigate the inconsistency (but not fully solve) the issue. Running expiry pager more often also might have disk I/O impact / tradeoff, where it would actually write deletes to disk. User should also be aware of expiry pager in tradeoff with all the other dispatcher / tasks.

        Frank, Dipti, Yaseen to discuss today (2012/11/02) outside of bug-scrub mtg and resolve.

        Show
        steve Steve Yen added a comment - For 2.0, options discussed in bug-scrub... Recommend to impacted customers to run expiry pager more often – e.g., once every 10 minutes for small/medium clusters. This can help mitigate the inconsistency (but not fully solve) the issue. Running expiry pager more often also might have disk I/O impact / tradeoff, where it would actually write deletes to disk. User should also be aware of expiry pager in tradeoff with all the other dispatcher / tasks. Frank, Dipti, Yaseen to discuss today (2012/11/02) outside of bug-scrub mtg and resolve.
        dipti Dipti Borkar made changes -
        Component/s view-engine [ 10060 ]
        Component/s documentation [ 10012 ]
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment - - edited

        Perhaps, I was not fully clear before.

        Tracking the expiration times in the indexes (values at the leaf nodes) would solve the issue for map view queries. True.
        However, we already have plenty of metadata tracked in the indexes (vbucket id for each value, 1024 bits/128 bytes bitmasks), that makes us lose some performance (deeper trees, smaller branching factor per tree node). At the moment, querying Apache CouchDB is faster than Couchbase Server (single node of course, to be fair).

        Now, for reduce views... Excluding values that were contributed by now-expired documents, means going down to all the leaf nodes to find out which values come from expired documents and which ones come from non-expired documents - then grab all the values from non-expired documents, applying reduce function against those values, and going up to the tree applying re-reduces until reaching the root. In other words, we would be doing not better than a linear scan, and defeating the whole purpose of intermediary reductions/efficiency for which CouchDB trees are known for.
        Basically doing this would, at the very best, give query response times of a few seconds (being very optimistic here) for any reasonably sized index (even less than than 1M items perhaps).

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - - edited Perhaps, I was not fully clear before. Tracking the expiration times in the indexes (values at the leaf nodes) would solve the issue for map view queries. True. However, we already have plenty of metadata tracked in the indexes (vbucket id for each value, 1024 bits/128 bytes bitmasks), that makes us lose some performance (deeper trees, smaller branching factor per tree node). At the moment, querying Apache CouchDB is faster than Couchbase Server (single node of course, to be fair). Now, for reduce views... Excluding values that were contributed by now-expired documents, means going down to all the leaf nodes to find out which values come from expired documents and which ones come from non-expired documents - then grab all the values from non-expired documents, applying reduce function against those values, and going up to the tree applying re-reduces until reaching the root. In other words, we would be doing not better than a linear scan, and defeating the whole purpose of intermediary reductions/efficiency for which CouchDB trees are known for. Basically doing this would, at the very best, give query response times of a few seconds (being very optimistic here) for any reasonably sized index (even less than than 1M items perhaps).
        steve Steve Yen made changes -
        Assignee Rahim Yaseen [ yaseen ] MC Brown [ mccouch ]
        Component/s documentation [ 10012 ]
        Component/s view-engine [ 10060 ]
        Hide
        mccouch MC Brown (Inactive) added a comment -

        The documentation has been updated at multiple points to highlight the inclusion of documents that may have expired, but not been fully deleted yet.

        Show
        mccouch MC Brown (Inactive) added a comment - The documentation has been updated at multiple points to highlight the inclusion of documents that may have expired, but not been fully deleted yet.
        mccouch MC Brown (Inactive) made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        kzeller kzeller added a comment -

        Added to Release Notes as:

        Couchbase Server does lazy expiration, that is, expired items are flagged as
        deleted rather than being immediately erased. Couchbase Server has
        a maintenance process that will periodically look through all information and erase expired items.
        This means expired items may still be indexed and appear in result sets of views. The workarounds
        are available here <ulink url="http://www.couchbase.com/docs/couchbase-devguide-2.0/about-ttl-values.html">
        About Document Expiration</ulink>.

        Show
        kzeller kzeller added a comment - Added to Release Notes as: Couchbase Server does lazy expiration, that is, expired items are flagged as deleted rather than being immediately erased. Couchbase Server has a maintenance process that will periodically look through all information and erase expired items. This means expired items may still be indexed and appear in result sets of views. The workarounds are available here <ulink url="http://www.couchbase.com/docs/couchbase-devguide-2.0/about-ttl-values.html"> About Document Expiration</ulink>.
        deepkaran.salooja Deepkaran Salooja made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        wayne Wayne Siu made changes -
        Component/s documentation-don't-use-put-in-doc-project [ 10012 ]

          People

          • Assignee:
            mccouch MC Brown (Inactive)
            Reporter:
            deepkaran.salooja Deepkaran Salooja
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes