Details
-
Improvement
-
Resolution: Done
-
Major
-
None
-
1
-
KV Sprint 2020-Oct
Description
Motivated by the following issue.
Magma tests currently crash when they are reinitialized with a different number of shards. The reason for this is that the mapping of vBuckets changes and magma uses the shard ID in the path of the data files. Whe a warmup happens after the nubmer of shards changes, the already existing shards pick up the old vBuckets but the new shards pick up none. In KV we maintain that vBuckets map to shards via modulus and for couchstore we can change the number of shards on restart as all the files exist in the same directory. We can't simply do this for magma as shards share a WAL and we'd lose data on restart (or have an overcomplicated recovery) if vBuckets get remapped to different shards.
Sarath Lakshman has suggested that we make the shards a fixed parameter for magma that cannot be changed after initialization. This is somewhat of an issue in KV as the number of shards we have maps directly to the number of flushers and bg fetchers that we can run simultaneously. If a customer were to upgrade from a machine with only a few CPUs (KV creates 1 shard per CPU) then we wouldn't be able to scale flushers or bg fetchers anymore. For magma we might be able to get away with not scaling KV flushers as they have to do less work than for couchstore (just write to WAL/memtable), but not being able to scale up bg fetchers after initialization may be problematic. In this case the customer would have to rebalance in/out instead of being able to do an offline upgrade to get out of this situation.
We can fix this problem in KV by decoupling the number of flushers/bg fetchers from the number of shards, which we could then make static for magma.
Attachments
Issue Links
For Gerrit Dashboard: MB-39745 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
152960,1 | MB-39745: Remove KVBucket::getROUnderlyingByShard() [5/n] | master | kv_engine | Status: NEW | 0 | 0 |
152961,1 | MB-39745: Remove KVBucket::getROUnderlying() [6/n] | master | kv_engine | Status: NEW | 0 | 0 |
152962,1 | MB-39745: Remove KVBucket::getOneROUnderlying() [7/n] | master | kv_engine | Status: NEW | 0 | 0 |
167165,1 | Revert "MB-39745: Don't over-allocate memory for vbAbortCompaction" | master | kv_engine | Status: NEW | -1 | -1 |
136376,6 | MB-39745: Remove KVShard from BGFetcher | master | kv_engine | Status: MERGED | +2 | +1 |
136377,10 | MB-39745: Add BgFetchers to EPBucket | master | kv_engine | Status: MERGED | +2 | +1 |
136378,9 | MB-39745: Scale bgFetchers independently of KVShards | master | kv_engine | Status: MERGED | +2 | +1 |
136651,10 | MB-39745: Make hpVbs a VBReadyQueue | master | kv_engine | Status: MERGED | +2 | +1 |
136652,14 | MB-39745: Notify flusher hpvbs from outside flusher | master | kv_engine | Status: MERGED | +2 | +1 |
136653,14 | MB-39745: Remove KVShard from Flusher | master | kv_engine | Status: MERGED | +2 | +1 |
136655,12 | MB-39745: Change KVBucket::getFlusher to getOneFlusher | master | kv_engine | Status: MERGED | +2 | +1 |
137409,5 | MB-39745: Remove BGFetcher::notifyBGEvent | master | kv_engine | Status: MERGED | +2 | +1 |
137426,13 | MB-39745: Move flushers to EPBucket | master | kv_engine | Status: MERGED | +2 | +1 |
137427,14 | MB-39745: Scale flushers independently of shards | master | kv_engine | Status: MERGED | +2 | +1 |
152890,7 | MB-39745: RO KVstore should return const RW KVStore | master | kv_engine | Status: MERGED | +2 | +1 |
152891,6 | MB-39745: Don't create RO KVStore | master | kv_engine | Status: MERGED | +2 | +1 |
152958,4 | MB-39745: Remove KVStoreRWRO and update set/take functions | master | kv_engine | Status: MERGED | +2 | +1 |
152959,3 | MB-39745: Remove ROStore | master | kv_engine | Status: MERGED | +2 | +1 |
152963,3 | MB-39745: Remove redundant RO store stat invocations | master | kv_engine | Status: MERGED | +2 | +1 |
152964,3 | MB-39745: Remove option in KVBucket::getKVStoreStats() | master | kv_engine | Status: MERGED | +2 | +1 |
152965,3 | MB-39745: Remove KVStore option from KVBucket:getKVStoreStat() | master | kv_engine | Status: MERGED | +2 | +1 |
152966,4 | MB-39745: Remove KStore::isReadWrite() | master | kv_engine | Status: MERGED | +2 | +1 |
152967,4 | MB-39745: Remove KVStore::isReadOnly() | master | kv_engine | Status: MERGED | +2 | +1 |
152968,4 | MB-39745: Remove KVStore::readOnly | master | kv_engine | Status: MERGED | +2 | +1 |
152969,5 | MB-39745: Remove CouchKVStore::makeReadOnlyStore() | master | kv_engine | Status: MERGED | +2 | +1 |
152970,6 | MB-39745: Remove Read/Write specific CouchKVStore ctors | master | kv_engine | Status: MERGED | +2 | +1 |
153125,4 | MB-39745: Remove KVStore::cachedDocCount | master | kv_engine | Status: MERGED | +2 | +1 |
153126,7 | MB-39745: Don't over-allocate memory for onDiskPrepareSize | master | kv_engine | Status: MERGED | +2 | +1 |
153127,6 | MB-39745: Don't over-allocate memory for spaceUsed | master | kv_engine | Status: MERGED | +2 | +1 |
153128,6 | MB-39745: Don't over-allocate memory for fileSize | master | kv_engine | Status: MERGED | +2 | +1 |
153129,5 | MB-39745: Don't over-allocate memory for deleteCount | master | kv_engine | Status: MERGED | +2 | +1 |
153130,7 | MB-39745: Don't over-allocate memory for vBucketStates | master | kv_engine | Status: MERGED | +2 | +1 |
153131,6 | MB-39745: Don't over-allocate memory for kvstoreRevList | master | kv_engine | Status: MERGED | +2 | +1 |
153166,5 | MB-39745: Don't over-allocate memory for vbAbortCompaction | master | kv_engine | Status: MERGED | +2 | +1 |
153167,5 | MB-39745: Don't over-allocate memory for vbCompactionRunning | master | kv_engine | Status: MERGED | +2 | +1 |
153168,8 | MB-39745: Manually size shards in VBucketMap | master | kv_engine | Status: MERGED | +2 | +1 |
153169,7 | MB-39745: Don't over-allocate memory for dvFileRevMap | master | kv_engine | Status: MERGED | +2 | +1 |
153934,6 | MB-39745: Move KVBucket::accessLog to Warmup | master | kv_engine | Status: MERGED | +2 | +1 |
156153,4 | MB-39745: Make KVStore::scan(ByID) const | master | kv_engine | Status: MERGED | +2 | +1 |
156156,4 | MB-39745: Make various KVStore functions const | master | kv_engine | Status: MERGED | +2 | +1 |
156157,5 | MB-39745: Make RocksDBKVStore::getVBHandle const | master | kv_engine | Status: MERGED | +2 | +1 |
156158,3 | MB-39745: Make KVStore scan(BySeqno) const | master | kv_engine | Status: MERGED | +2 | +1 |
156159,3 | MB-39745: CouchKVStore open functions should be const | master | kv_engine | Status: MERGED | +2 | +1 |
156192,2 | MB-39745: Make KVStore::makeFileHandle const | master | kv_engine | Status: MERGED | +2 | +1 |
156193,2 | MB-39745: Make various CouchKVStore functions const | master | kv_engine | Status: MERGED | +2 | +1 |
156194,2 | MB-39745: Make KVStore::initByIdScanContext const | master | kv_engine | Status: MERGED | +2 | +1 |
156195,2 | MB-39745: Make KVStore::readVBState functions const | master | kv_engine | Status: MERGED | +2 | +1 |
156196,2 | MB-39745: Make KVStore::initBySeqnoScanContext const | master | kv_engine | Status: MERGED | +2 | +1 |
156197,4 | MB-39745: Make trivial KVStore::get functions const | master | kv_engine | Status: MERGED | +2 | +1 |
156198,3 | MB-39745: Make KVStore::getWithHeader() const | master | kv_engine | Status: MERGED | +2 | +1 |
156199,3 | MB-39745: Make KVStore::getMulti() const | master | kv_engine | Status: MERGED | +2 | +1 |
156200,3 | MB-39745: KVStore::getStorageProperties() should be const | master | kv_engine | Status: MERGED | +2 | +1 |
156201,3 | MB-39745: KVStore::getCollectionsManifest() should be const | master | kv_engine | Status: MERGED | +2 | +1 |
156202,3 | MB-39745: KVStore stats functions should be const | master | kv_engine | Status: MERGED | +2 | +1 |
156203,3 | MB-39745: Make KVStore::getCollectionsStats() const | master | kv_engine | Status: MERGED | +2 | +1 |
156206,2 | MB-39745: Make various KVStore functions const | master | kv_engine | Status: MERGED | +2 | +1 |
156207,3 | MB-39745: Make backfill functions use const KVStore | master | kv_engine | Status: MERGED | +2 | +1 |
156208,3 | MB-39745: Use RW KVStore where appropriate | master | kv_engine | Status: MERGED | +2 | +1 |
166720,3 | Revert "MB-39745: Don't over-allocate memory for vBucketStates" | master | kv_engine | Status: ABANDONED | -2 | -1 |