[FTS] high disk usage during initial indexing in DGM scenario

.. and also weird behavior.

The test:  Initial indexing fo 40M dataset, single FTS node, 32GB RAM, various quotas.
First run was with very high FTS quota to measure the max RAM fts would use when initial indexing.
The experiment shows that on single FTS node, to support its maximum indexing throughput of 40 MB/sec  the fts uses 10GB of RAM

Having that in mind, I ran few tests with quota lower that 10GB considering that tests as "DGM" kind of scenarios.

The behavior is: the higher the quota within those 10GB the slower the indexing and the more disk space it uses.

First, the indexing duration:

Once we get into DGM, the indexing throughput drops 4-5 times. And the higher the quota the slower it gets:

Same is with disk size, but even on larger scale:

Here is comparison of disk usage pattern with various quota sizes:

Full comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-1614_build_index_ec7d&label=1%20GB&snapshot=fts_600-1614_build_index_3fc7&label=2%20GB&snapshot=fts_600-1614_build_index_f2fc&label=4%20GB&snapshot=fts_600-1614_build_index_7a37&label=6%20GB&snapshot=fts_600-1614_build_index_82f5&label=%3E10%20GB

Its getting fun when comparing how system behaves with 10GB and 8GB quota assuming that we just 20% less that max required:

Especially because it uses the same amount of memory

Logs: