Details
-
Improvement
-
Resolution: Done
-
Major
-
None
-
1
-
KV Sprint 2020-Oct
Description
Motivated by the following issue.
Magma tests currently crash when they are reinitialized with a different number of shards. The reason for this is that the mapping of vBuckets changes and magma uses the shard ID in the path of the data files. Whe a warmup happens after the nubmer of shards changes, the already existing shards pick up the old vBuckets but the new shards pick up none. In KV we maintain that vBuckets map to shards via modulus and for couchstore we can change the number of shards on restart as all the files exist in the same directory. We can't simply do this for magma as shards share a WAL and we'd lose data on restart (or have an overcomplicated recovery) if vBuckets get remapped to different shards.
Sarath Lakshman has suggested that we make the shards a fixed parameter for magma that cannot be changed after initialization. This is somewhat of an issue in KV as the number of shards we have maps directly to the number of flushers and bg fetchers that we can run simultaneously. If a customer were to upgrade from a machine with only a few CPUs (KV creates 1 shard per CPU) then we wouldn't be able to scale flushers or bg fetchers anymore. For magma we might be able to get away with not scaling KV flushers as they have to do less work than for couchstore (just write to WAL/memtable), but not being able to scale up bg fetchers after initialization may be problematic. In this case the customer would have to rebalance in/out instead of being able to do an offline upgrade to get out of this situation.
We can fix this problem in KV by decoupling the number of flushers/bg fetchers from the number of shards, which we could then make static for magma.