Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-20197

No purge for deleted items with full eviction

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • None
    • 4.1.1
    • couchbase-bucket
    • None
    • Cluster of 3 nodes 4VCPU / 4GB RAM
    • Untriaged
    • Ubuntu 64-bit
    • Unknown

    Description

      No metadata purge is detected if "full ejection" = true on bucket even if manual compaction is called.
      How to reproduce:
      1. Establish a new cluster: 3 nodes of 4.1.X (last my try was on 6074) with 4 VCPU / 4 GB RAM
      2. Go to UI -> Settings -> Auto-compaction and set "Database Fragmentation" = 10% or 1Mb,
      "Metadata Purge Interval" = 0.04 and save
      3. Create a default bucket (memory = 128M per node, 1 replica, full ejection, io-priority=high, view index replicas, flush = enable)
      4. Insert 100K empty docs with keys formed like "key = 'a:' + System.currentTimeMillis() + ':' + System.currentTimeMillis() " with TTL=1 minute (for example: "a:1468852435599:1468852435599")
      5. Wait 1+ minutes (10 min to be absolutely sure) after last insertion.
      6. Go to UI -> Data Buckets -> Default. You should see now "Item count" = 0 for "default" bucket.
      7. Now, press "Documents" and wait a while. There will be "There are currently no documents in this bucket corresponding to the search criteria" message on UI (because high--hardware-configuration and just 100K items)
      8. Now, take a look at babysitter.log (tail -f /opt/couchbase/var/lib/couchbase/logs/babysitter.log -n 100). You will see messages like this:

      memcached<0.75.0>: 2016-07-19T11:18:01.299591Z WARNING (default) Warning: failed to fetch data from database, vBucket=103 key=0:1468852395575:1468852395575 error=document not found [none]
      memcached<0.75.0>: 2016-07-19T11:18:01.299787Z WARNING (default) Warning: failed to fetch data from database, vBucket=240 key=0:1468852397352:1468852397352 error=document not found [none]
      memcached<0.75.0>: 2016-07-19T11:18:01.299998Z WARNING (default) Warning: failed to fetch data from database, vBucket=261 key=0:1468852400100:1468852400100 error=document not found [none]
      memcached<0.75.0>: 2016-07-19T11:18:01.300190Z WARNING (default) Warning: failed to fetch data from database, vBucket=86 key=0:1468852402016:1468852402016 error=document not found [none]
      

      This is "very tricky" demonstration of non-purged items, that happens because of https://issues.couchbase.com/browse/MB-19758 / MB-16663. As we can see from http://review.couchbase.org/gitweb?p=ep-engine.git;a=blobdiff;f=src/couch-kvstore/couch-kvstore.cc;h=bb29402349b4bb4d51e70ad6645ca169dffd2e97;hp=7c856ad9b2375bffe080a54f6c6326d868d3ed30;hb=bf22617cb39d333e548aef177f5c7b8538193331;hpb=191fda08d638dae93f5dfcd1146340ffefae87ce option COUCHSTORE_NO_OPTIONS should be replaced with COUCHSTORE_NO_DELETES to avoid listing of deleted/purged items. This was done for 4.5, but was not for 4.1.

      [disclaimer: i broke a couple of spears against Dave Rigby's breastplate trying to backport this to 4.1, but he fought till the death, so this remained unfixed in 4.1. Probably all this fight was kinda "divine intervention" (mmm, i love this phrase!) that was needed to find this "no purge" bug, but Dave, i really think now it's time to fix it, and, as you know, i'm ready to backport a patch and test by myself; all i need is your potential approval for Gerrit ;)]

      9. Now, go to UI->Data buckets -> default and press "Compact" and wait until it's done.
      10. Once again check babysitter.log You will see messages like in p.8 again.

      There is a forum topic with more details: https://forums.couchbase.com/t/4-1-x-issue-no-purge-for-128mb-bucket-full-ejection-even-for-manual-compaction/9155

      You can find there simple insertion code (java) and different example of working purge if "full ejection" = false

      Attachments

        1. 4.1.1.GA.F.tar.bz2
          1.97 MB
        2. 4.1.1.GA.V.tar.bz2
          735 kB
        3. testing-no-purge.pdf
          67 kB
        4. testing-no-purge-v2.pdf
          84 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            manu Manu Dhundi (Inactive)
            egrep Georgy Repin
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty