Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47604

NexusKVStore implementation

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Major
    • 7.1.0
    • 7.1.0
    • couchbase-bucket
    • None
    • 1
    • KV-Engine Sprint 2021 August, KV-Engine-Sept-21, KV 2021-Oct-21, KV 2021-Nov, KV 2021-Dec, KV 2022-Jan, KV 2022-Feb, KV March-22

    Description

      NOTE BEFORE CLOSING

      • Clean up less useful variants (i.e. places where we run magma as primary and couchstore as secondary to test Nexus as well as couchstore primary, magma secondary) for the sake of keeping CV fast.
      • Either remove the moving expiration time point code during compaction or create an MB to track removal. MB-51511

      It would be good to create some additional framework around KVStore such that we could run two in parallel (of any type) and compare the results of any operation against one against the results against another. They should generally (always?) be the same. There are some interesting cases where things work differently between the current KVStores (such as compaction) that may prove less than trivial in doing so. Such a KVStore should be a drop in replacement for any other KVStore so that we could even run this in QE tests. This may required some refactoring such that we separate the API of KVStore from the common implementations that live there.

      Brief overview on NexusKVStore - https://docs.google.com/document/d/1bbak-rTblXY8AjMYRsxcY5oJrhwMJs5y_61R8my4Mko/edit#

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-47762 On disk manifest uid gets reset when we overwrite a logically deleted but not yet purged document

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-47811 MagmaKVStore - Stat doc not recreated after collection resurrection

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48172 MagmaKVStore skipping rollback callback invocations for system event items

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48504 MagmaKVStore does not check snapshot validity

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-48182 Make couch-kvstore seqno midpoint rollback optimization configurable

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-47991 Magma SyncDelete prepare has metadata deleted bit set

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48128 Fail MagmaKVStore::getCollectionsManifest if NotExists

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48447 After rollback to zero magma appears to return collection stats from pre-rollback

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48660 Magma rollback points differ to couchstore with checkpoint every batch

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-47604
          # Subject Branch Project Status CR V
          158441,11 MB-47604: Separate public API of KVStore master kv_engine Status: MERGED +2 +1
          158442,11 MB-47604: Add initial NexusKVStore implentation master kv_engine Status: MERGED +2 +1
          158477,3 MB-47604: Remove RocksDBKVStore::prepareForDeduplication master kv_engine Status: MERGED +2 +1
          158478,4 MB-47604: Remove Magma and Rocks snapshotStats variant master kv_engine Status: MERGED +2 +1
          158488,27 MB-47604: Add NexusKVStore config master kv_engine Status: MERGED +2 +1
          158592,5 MB-47604: Move KVStoreConfig creation to static function master kv_engine Status: MERGED +2 +1
          158593,5 MB-47604: Make backend non-optional KVStoreConfig parameter master kv_engine Status: MERGED +2 +1
          158594,11 MB-47604: Create secondary KVStore in NexusKVStore master kv_engine Status: MERGED +2 +1
          158595,9 MB-47604: Remove StorageProperties::ConcurrentWriteCompact master kv_engine Status: MERGED +2 +1
          158596,11 MB-47604: Implement NexusKVStore::getStorageProperties() master kv_engine Status: MERGED +2 +1
          158597,11 MB-47604: Implement trivial NexusKVStore functions master kv_engine Status: MERGED +2 +1
          158598,5 MB-47604: Split KVStoreIFace::commit in two - delete me master kv_engine Status: ABANDONED 0 0
          158649,5 MB-47604: Unify single threaded test backend config master kv_engine Status: MERGED +2 +1
          158672,7 MB-47604: Make it possible to swap Flush::manifest master kv_engine Status: MERGED +2 +1
          158828,8 MB-47604: Make Nexus KVStores write to different directories master kv_engine Status: MERGED +2 +1
          158830,3 MB-47604: Add test variants master kv_engine Status: ABANDONED 0 -1
          159001,3 MB-47604: Split up comparison of Collections::KVStore::Manifest master kv_engine Status: MERGED +2 +1
          159002,12 MB-47604: Add flush functionality to NexusKVStore master kv_engine Status: MERGED +2 +1
          159084,9 MB-47604: Add logging to interesting NexusKVStore operations master kv_engine Status: MERGED +2 +1
          159142,8 MB-47604: Add StorageProperties::PrepareCounting master kv_engine Status: MERGED +2 +1
          159143,9 MB-47604: Add NexusKVStore::snapshotVBucket functionality master kv_engine Status: MERGED +2 +1
          159144,13 MB-47604: Add locking to flushing and gets in NexusKVStore master kv_engine Status: MERGED +2 +1
          159145,14 MB-47604: Add NexusKVStore::get functionality master kv_engine Status: MERGED +2 +1
          159207,3 MB-47604: Add KVStore collections meta ostr functions master kv_engine Status: MERGED +2 +1
          159208,5 MB-47604: Move endTransaction to KVStoreIface master kv_engine Status: MERGED +2 +1
          159234,12 MB-47604: Add NexusPersistenceCallbacks to compare results master kv_engine Status: MERGED +2 +1
          159248,5 MB-47604: Add magma/couchstore nexus variant master kv_engine Status: MERGED +2 +1
          159249,12 MB-47604: Add NexusKVStore getMulti (bgFetch) support master kv_engine Status: MERGED +2 +1
          159290,9 MB-47604: Add Nexus variant to KVStoreParam test master kv_engine Status: MERGED +2 +1
          159294,13 MB-47604: Implement NexusKVStore::getRange master kv_engine Status: MERGED +2 +1
          159533,6 MB-47604: Rename doPostFlushSanityChecks master kv_engine Status: MERGED +2 +1
          159534,26 MB-47604: Add support for compaction to NexusKVStore master kv_engine Status: MERGED +2 +1
          159714,4 MB-47604: Add storage properties for compaction stale item callbacks master kv_engine Status: MERGED +2 +1
          160231,6 MB-47604: Add config for expiration from start of compaction master kv_engine Status: MERGED +2 +1
          160232,7 MB-47604: Set checkpoint every batch magma config for nexus master kv_engine Status: MERGED +2 +1
          160233,7 MB-47604: Make rollback test param consistent with other tests master kv_engine Status: MERGED +2 +1
          160236,13 MB-47604: Add rollback support to NexusKVStore master kv_engine Status: MERGED +2 +1
          160237,10 MB-47604: Implement NexusKVStore::listPersistedVbuckets() master kv_engine Status: MERGED +2 +1
          160238,6 MB-47604: Implement getCached/PersistedVBucketState() for Nexus master kv_engine Status: MERGED +2 +1
          160239,12 MB-47604: Implement NexusKVStore::getItemCount() master kv_engine Status: MERGED +2 +1
          160656,5 MB-47604: Implement NexusKVStore::getLastPersistedSeqno() master kv_engine Status: MERGED +2 +1
          160657,11 MB-47604: Implement getCollectionsMeta functions for Nexus master kv_engine Status: MERGED +2 +1
          160658,7 MB-47604: Implement NexusKVStore::getAllKeys() master kv_engine Status: MERGED +2 +1
          160659,9 MB-47604: Make KVStore::getPersistedStats() const master kv_engine Status: MERGED +2 +1
          160660,6 MB-47604: Make KVStore::getDroppedCollections() const master kv_engine Status: MERGED +2 +1
          160886,2 MB-47604: Expire all items from start of compaction master kv_engine Status: MERGED +2 +1
          160983,4 MB-47604: Protect ScanContext callbacks and make getters virtual master kv_engine Status: MERGED +2 +1
          160984,8 MB-47604: Implement NexusKVStore::scan() master kv_engine Status: MERGED +2 +1
          161189,2 Revert "MB-47604: Set checkpoint every batch magma config for nexus" master kv_engine Status: MERGED +2 +1
          161194,6 Recommit MB-47604: Set checkpoint every batch magma config for nexus master kv_engine Status: MERGED +2 +1
          161678,2 MB-47604: Enable some skipped nexus tests master kv_engine Status: MERGED +2 +1
          161698,3 MB-47604: Don't take vb state lock when warming up prepares master kv_engine Status: MERGED +2 +1
          161752,3 MB-47604: Fix DelVBucketConcurrentOperations test for nexus master kv_engine Status: MERGED +2 +1
          161765,8 MB-47604: Fail CouchKVStore compaction if no vbstate master kv_engine Status: MERGED +2 +1
          161849,4 MB-47604: Implement NexusKVStore::getStat master kv_engine Status: MERGED +2 +1
          162178,4 MB-47604: Implement NexusKVStore::makeFileHandle master kv_engine Status: MERGED +2 +1
          162186,5 MB-47604: Implemented NexusKVStore::getManifestUid master kv_engine Status: MERGED +2 +1
          162191,4 MB-47604: Implement NexusKVStore::getCollectionStats master kv_engine Status: MERGED +2 +1
          162194,4 MB-47604: Implement NexusKVStore::getBySeqno master kv_engine Status: MERGED +2 +1
          162346,7 MB-47604: Rename max_purged_seqno to rollbackPurgeSeqno master kv_engine Status: MERGED +2 +1
          162347,5 MB-47604: Only compare value of item if it exists master kv_engine Status: MERGED +2 +1
          162359,6 MB-47604: Remove redundant setting of purge stats in vbstate master kv_engine Status: MERGED +2 +1
          162360,5 MB-47604: Fix NexusKVStore::getAllKeys master kv_engine Status: MERGED +2 +1
          162588,6 MB-47604: Set magma checkpoint interval for nexus master kv_engine Status: MERGED +2 +1
          163011,3 MB-47604: Rename CompactionContext::maybeUpdatePurgeSeqno master kv_engine Status: MERGED +2 +1
          163012,24 MB-47604: Implicit compaction support for NexusKVStore master kv_engine Status: MERGED +2 +1
          163092,8 MB-47604: Have CompactionContext update rollbackPurgeSeqno master kv_engine Status: MERGED +2 +1
          163093,1 MB-47604: Make CompactionContext class master kv_engine Status: ABANDONED 0 -1
          163170,9 MB-47604: Rename MagmaKVStore::makeCompactionContext master kv_engine Status: MERGED +2 +1
          163171,1 MB-47604: Build CompactionContext via KVStore master kv_engine Status: ABANDONED 0 0
          163186,14 MB-47604: Move rollbackPurgeSeqno update into new ctx class master kv_engine Status: MERGED +2 +1
          163188,13 MB-47604: Move MagmaDbStats purgeSeqno update to MagmaRollbackPurgeSeqnoCtx master kv_engine Status: MERGED +2 +1
          163189,10 MB-47604: Move VBucket (in-mem) purgeSeqno update to PurgedItemContext master kv_engine Status: MERGED +2 +1
          163193,5 MB-47604: Pass CompactionConfig by value and use std::move master kv_engine Status: MERGED +2 +1
          163447,5 MB-47604: Move CompactionCtx::purgedItem to PurgedItemCtx master kv_engine Status: MERGED +2 +1
          163478,5 MB-47604: Check secondary callbacks not made by primary for getRange master kv_engine Status: MERGED +2 +1
          163588,27 MB-47604: Support concurrent flush and compaction in Nexus master kv_engine Status: MERGED +2 +1
          163718,5 MB-47604: Move cache slot logic from getLock to separate function master kv_engine Status: MERGED +2 +1
          163719,7 MB-47604: Shard skipGetWithHeaderChecksForRollback per vBucket master kv_engine Status: MERGED +2 +1
          163720,5 MB-47604: Print manifest rather than uid when incorrect after flush master kv_engine Status: MERGED +2 +1
          167003,2 MB-47604: Add collectionsPurged compaction stat [1/3] master kv_engine Status: MERGED +2 +1
          167004,5 MB-47604: [Nexus] Skip compaction checks if purging collections [2/3] master kv_engine Status: MERGED +2 +1
          167005,5 MB-47604: Move NexusKVStore purgeSeqno for explicit compactions [3/3] master kv_engine Status: MERGED +2 +1
          167197,2 MB-47604: [Nexus] Add config to disable implicit compaction master kv_engine Status: MERGED +2 +1
          168515,2 MB-47604: Enabled disabled nexus tests master kv_engine Status: MERGED +2 +1
          168516,2 MB-47604: Remove unnecessary Nexus test variants master kv_engine Status: MERGED +2 +1
          171302,2 MB-47604: Enable concurrent flush and compaction for nexus master kv_engine Status: MERGED +2 +1

          Activity

            People

              ben.huddleston Ben Huddleston
              ben.huddleston Ben Huddleston
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty