Description
Currently we don't remove values upgraded to chronicle from ns_config (due to it being non-trivial to orchestrate in a race-free manner). Separately, many heavy REST APIs call ns_config:get() for each request. With move to chronicle, such APIs will also call chronicle_kv:get_full_snapshot() (this bit is something that could and should eventually be changed, but that's how things are at the moment). Ultimately, users that upgraded their clusters will pay double the price compared to users running fresh clusters. Similarly, in places where metadata needs to be synchronized across cluster, we'll continue paying the price of synchronizing hefty ns_config in addition to synchronizing chronicle.
A couple of things we might do:
1. Clean up ns_config on upgrade.
2. Minimize the use of get_full_snapshot() to absolute minimum.
There's probably more that we can do, but that's what comes to mind at the moment.
We should also definitely run some tests on extreme configurations. And it's critical to remember that fresh clusters may not expose some of the behaviors that upgraded clusters will.
Attachments
For Gerrit Dashboard: MB-44838 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
148732,1 | MB-44838 Remove buckets from ns_config post-upgrade | master | ns_server | Status: ABANDONED | -2 | 0 |
150332,6 | MB-44838 do not use chronicle_kv:get_full_snapshot() when fetching | master | ns_server | Status: MERGED | +2 | +1 |
151932,8 | MB-44838 ns_config_log should tolerate buckets key being deleted | master | ns_server | Status: MERGED | +2 | +1 |
151933,8 | MB-44838 delete keys moved to chronicle from ns_config one minute | master | ns_server | Status: MERGED | +2 | +1 |