Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.0
-
Untriaged
-
Yes
-
KV Spint 2020-March
Description
In full eviction when using an EP bucket we can end up with a current item count higher than the number of items in the bucket. This is due to the fact that the current number of items its tracked by EPVBucket::onDiskTotalItems in full eviction.
The onDiskTotalItems counter's value can become greater than the number of items on disk due to a bug in VBucket::deletedOnDiskCbk(), where we don't decrement the number of items on disk when we can't find a value for the queued item in the hash table. This was fine before durability as we wouldn't have removed the value till VBucket::deletedOnDiskCbk() had been called. However, now with durability we remove the item with the same key from the hash table during the commit phase of a SyncWrite. Thus, if the commit happens in the hash table before VBucket::deletedOnDiskCbk() is called for the delete, we won't find an item and then decrement the number of items on disk.
void VBucket::deletedOnDiskCbk(const Item& queuedItem, bool deleted) { |
auto res = ht.findItem(queuedItem);
|
auto* v = res.storedValue;
|
// Delete the item in the hash table iff: |
// 1. Item is existent in hashtable, and deleted flag is true |
// 2. rev seqno of queued item matches rev seqno of hash table item |
if (v && v->isDeleted() && (queuedItem.getRevSeqno() == v->getRevSeqno())) { |
EP_LOG_ERR("VBucket::deletedOnDiskCbk deleteStoredValue() key:{}", |
queuedItem.getKey().c_str());
|
bool isDeleted = deleteStoredValue(res.lock, *v); |
if (!isDeleted) { |
throw std::logic_error( |
"deletedOnDiskCbk:callback: " |
"Failed to delete key with seqno:" + |
std::to_string(v->getBySeqno()) + "' from bucket " + |
std::to_string(res.lock.getBucketNum()));
|
}
|
|
/** |
* Deleted items are to be added to the bloomfilter,
|
* in either eviction policy.
|
*/
|
addToFilter(queuedItem.getKey());
|
}
|
|
if (deleted) { |
++stats.totalPersisted;
|
++opsDelete;
|
|
/** |
* MB-30137: Decrement the total number of on-disk items. This needs to
|
* be done to ensure that the item count is accurate in the case of full
|
* eviction. We should only decrement the counter for committed (via
|
* mutation or commit) items as we only increment for these.
|
*/
|
if (v && queuedItem.isCommitted()) { |
decrNumTotalItems();
|
}
|
}
|
...
|
}
|
Incorrect item count on the active we should only have one item like we do on the replicas:
Attachments
Issue Links
For Gerrit Dashboard: MB-38197 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
123293,3 | MB-38197 | master | kv_engine | Status: ABANDONED | 0 | -1 |
123472,4 | MB-38197: On-disk #items incorrect after Delete + SyncAdd | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
123523,3 | MB-38197: On-disk #items incorrect after SyncWrite + Delete | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
123597,1 | Merge remote-tracking branch 'couchbase/mad-hatter' | master | kv_engine | Status: ABANDONED | 0 | -1 |
123632,1 | Merge remote-tracking branch 'couchbase/mad-hatter' | master | kv_engine | Status: MERGED | +2 | +1 |