Ok, I'm glad we caught this early.
Not only do I see higher memory consumption that's causing VERY SLOW indexing progress, some of the DCP mutations aren't making it into FTS at all.
I've collected heap profiles upon index build, and here's how it looked, almost completed dominated by gocbcore's dialMemdClient -

I've found the change within gocbcore that's responsible for this (sorry James Lee, not-so-good news from FTS on your change)..
commit aa0eefea765225169e0bce8e6026e70be6aab364
|
Author: James Lee <james.lee@couchbase.com>
|
Date: Mon Feb 8 16:58:25 2021 +0000 GOCBC-1056 Reduce CPU usage
|
|
Motivation
|
----------
|
After doing some performance testing of cbbackupmgr we noticed that we
|
were spending a significant amount of CPU time in gocbcore. We'd like to
|
reduce this as much as possible to free up CPU time for cbbackupmgr.
|
|
Changes
|
-------
|
1) Avoid unnecessary allocations of DCP packets/TCP write buffers
|
2) Avoid unnecessary branching using if statements in the golden path
|
where a switch statement would suffice.
|
3) Rewrote the memdopmap implementation to use a map structure instead
|
of a doubly linked list structure. This avoids linear traversal for
|
each request which depending on the scenario could result in
|
not-insignificant overhead.
|
4) Added the ability to disable buffer acknowledgement via a option
|
using the DCP agent.
|
5) Use a 20MB buffered reader on each memcached connection, the same
|
size as used by KV engine.
|
...
|
On reverting the change above I see the index progress is quick and back to how it was before we upgraded to gocbcore@9.1.3. I also grabbed the heap profile upon index build this time around and this is how it looked -

Brett Lawson Charles Dixon what's the play here? Should FTS ..
- Wait for a quick fix within gocbcore and an immediate release after
- Revert aa0eefe within gocbcore and an immediate release after (and perhaps try adding the fixed/corrected change back later)
- Move FTS back to gocbcore@9.1.2 until a fix is made to this and then skip 9.1.3 altogether.
Please be advised that we'll need to unblock QE as quickly as possible.
Higher memory consumption's the reason - the app herder has floored the brake.