Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38197

EP buckets in full eviction mode can end up with incorrect high item count

    XMLWordPrintable

Details

    • Untriaged
    • Yes
    • KV Spint 2020-March

    Description

      In full eviction when using an EP bucket we can end up with a current item count higher than the number of items in the bucket. This is due to the fact that the current number of items its tracked by EPVBucket::onDiskTotalItems in full eviction.
      The onDiskTotalItems counter's value can become greater than the number of items on disk due to a bug in VBucket::deletedOnDiskCbk(), where we don't decrement the number of items on disk when we can't find a value for the queued item in the hash table. This was fine before durability as we wouldn't have removed the value till VBucket::deletedOnDiskCbk() had been called. However, now with durability we remove the item with the same key from the hash table during the commit phase of a SyncWrite. Thus, if the commit happens in the hash table before VBucket::deletedOnDiskCbk() is called for the delete, we won't find an item and then decrement the number of items on disk.

      void VBucket::deletedOnDiskCbk(const Item& queuedItem, bool deleted) {
          auto res = ht.findItem(queuedItem);
          auto* v = res.storedValue;
          // Delete the item in the hash table iff:
          //  1. Item is existent in hashtable, and deleted flag is true
          //  2. rev seqno of queued item matches rev seqno of hash table item
          if (v && v->isDeleted() && (queuedItem.getRevSeqno() == v->getRevSeqno())) {
              EP_LOG_ERR("VBucket::deletedOnDiskCbk deleteStoredValue() key:{}",
                         queuedItem.getKey().c_str());
              bool isDeleted = deleteStoredValue(res.lock, *v);
              if (!isDeleted) {
                  throw std::logic_error(
                          "deletedOnDiskCbk:callback: "
                          "Failed to delete key with seqno:" +
                          std::to_string(v->getBySeqno()) + "' from bucket " +
                          std::to_string(res.lock.getBucketNum()));
              }
       
              /**
               * Deleted items are to be added to the bloomfilter,
               * in either eviction policy.
               */
              addToFilter(queuedItem.getKey());
          }
       
          if (deleted) {
              ++stats.totalPersisted;
              ++opsDelete;
       
              /**
               * MB-30137: Decrement the total number of on-disk items. This needs to
               * be done to ensure that the item count is accurate in the case of full
               * eviction. We should only decrement the counter for committed (via
               * mutation or commit) items as we only increment for these.
               */
              if (v && queuedItem.isCommitted()) {
                  decrNumTotalItems();
              }
          }
      ...
      }
      

      Incorrect item count on the active we should only have one item like we do on the replicas:

      Attachments

        1. screenshot-1.png
          screenshot-1.png
          158 kB
        2. test.txt
          70 kB
        3. test-no-info.txt
          39 kB

        Issue Links

          For Gerrit Dashboard: MB-38197
          # Subject Branch Project Status CR V

          Activity

            People

              sumedh.basarkod Sumedh Basarkod (Inactive)
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty