Change default Reader/Writer setting to "Disk I/O Optimized" (and maybe rebrand?)

Background

Disk I/O Optimized thread counts for Reader and Writer threads were introduced in 6.5.0 alongside SyncWrites, to improve both throughput and latency of operations requiring disk read/writes; by increasing the number of threads created for performing Reader / Writer tasks.

Recall the current (7.2.0) thread counts for Default and "Disk I/O Optimized"[1] modes, both of which are a function of the number of logical CPU cores available to the node (NCPUS):

• Default
  • Readers: NCPU threads, min 4 max 16.
  • Writers: Fixed at 4 threads.
• Disk I/O Optimized
  • Readers: NCPU threads, min 4 max 128.
  • Writers: NCPU threads, min 4 max 128.

[1]: The Disk I/O Optimized numbers are planned to increase as part of MB-57325
Note that the default setting for these threads remained the same as it was pre-6.5.0; as we saw some regression in specific front-end heavy workloads with little bgFetches - i.e. the extra CPU cost of persisting items to disk faster (more, smaller batches) took CPU resource away from the front-end threads. TODO: get reference / details of those regressions. As such, "Disk I/O Optimized" is an opt-in setting which users have to change themselves.

Proposal

Change the OOTB default Reader & Writer thread setting to "Disk I/O Optimized" for newly-created clusters.

While currently we default to the pre-6.5.0 configuration, I would assert that most workloads would benefit from having Disk I/O Optimized actually be the "default", automatically chosen option, compared to what I believe is a small minority of use-cases where Default actually gives better performance (aka I think the number of regressions we'd see from changing the OOTB default to "Disk I/O Optimized" is significantly smaller than the number of progressions our customers would see:

• Writers - the writer thread count is only 4 threads when Default is selected; this is insufficient to saturate (or even come close to) typical disks users use in contemporary environements - particulary EBS-style network-attached disks with can only reach their advertised throughputs when queue depths of the order of 10s is generated.
• Readers - the reader thread count is capped at 16 instead of 128 when Default is selected. On first glance this might seem like less of an issue (i.e. only nodes with CPU counts greater than 16 would see any change in behaviour between Default & Disk IO Optimized); however when considering MB-57325 ("Increase number of threads created for "Disk I/O Optimized" reader/writers") we will soon have "Disk I/O Optimized" reader threads on a 16 core machine creating 32 or potentially 64 reader threads; a significant difference from the 16 which Default creates. Again this is due to the need to increase the disk queue depth to many 10s of IOPS to be close to saturating EBS-style disks.

Note that reader threads are arguably the bigger issue here given they are "blocking" operations - one cannot complete say a BGFetch operation until the actual disk read completes. Compared this to (non-Sync)Writes which return to the user as soon as the mutation is queued in RAM so the front-end operation isn't blocking on the IO to occur. Additionally Magma has additional background threads performing writes for SST files / compaction (only the WAL write is done in the ep-engine "Writer" thread) and hence there's additional threads writing to drive some additional write queue depth.

Naming considerations

If we make "Disk I/O Optimized" the OOTB default value, then having the other, legacy setting named "Default" is potentially confusing - it's not actually the default anymore!

I would therefore suggest we rename in the UI "Default" to something else. Naming is hard and all that, but possible suggestions would be:

• Legacy (implying the old setting before we improved things - also has somewhat negative connotations so encourages users to not change back to it / if on
• Conservative (possibly more meaningful, suggests this one has "less" of something / uses fewer resources but in turn doesn't perform as well. Again, a negative connotation to tech people who want "best / fastest / aggressive"
  ... further suggestions welcome...