metakv performance measurements

Dave Finlay This is the MB I mentioned to you in Slack 2021-02-10 Jeelan Poola asked me to open to collect metakv performance measurements I did for MB-43285 in an MB for ns_server to look at.

Summary of findings (these were all from GSI delete tokens, which are ~200 bytes each):

1. 1us = per-tombstone incremental cost of stepping over delete token tombstones (in metakv.ListAllChildren("/indexing/ddl/commandToken/delete/") API)
2. 1ms = per delete token incremental cost of listing live delete tokens (from same API call)
3. 20ms = per delete token cost of retrieving one delete token (metakv.Get() API)

Thus tombstones are 3 orders of magnitude faster than listing a token and 4 orders of magnitude faster than retrieving a token.

Discussion with you today uncovered that metkav.ListAllChildren() actually retrieves not just the list of keys but also all the values of those keys (i.e. all the tokens, in GSI parlance). GSI code evidently did not realize this, as it has been first listing all the tokens via metakv.ListAllChildren() and then iterating over the paths returned by that call to retrieve each token one by one via metakv.Get(). In fact the tokans are already present as the Value []byte slices in the entries returned by metakv.ListAllChildren(). (I have verified this empirically.)

For #1-2 I will attach graphs and the raw data point files. #3 is derived from the experiment for #2 (details below).