Abhinav Dangeti added a comment - 18/Mar/21 3:48 PM Higher memory consumption's the reason - the app herder has floored the brake.

Abhinav Dangeti added a comment - 18/Mar/21 5:06 PM - edited Ok, I'm glad we caught this early. Not only do I see higher memory consumption that's causing VERY SLOW indexing progress, some of the DCP mutations aren't making it into FTS at all. I've collected heap profiles upon index build, and here's how it looked, almost completed dominated by gocbcore's dialMemdClient - I've found the change within gocbcore that's responsible for this (sorry James Lee , not-so-good news from FTS on your change).. commit aa0eefea765225169e0bce8e6026e70be6aab364 Author: James Lee <james.lee @couchbase .com> Date: Mon Feb 8 16 : 58 : 25 2021 + 0000 GOCBC- 1056 Reduce CPU usage Motivation ---------- After doing some performance testing of cbbackupmgr we noticed that we were spending a significant amount of CPU time in gocbcore. We'd like to reduce this as much as possible to free up CPU time for cbbackupmgr. Changes ------- 1 ) Avoid unnecessary allocations of DCP packets/TCP write buffers 2 ) Avoid unnecessary branching using if statements in the golden path where a switch statement would suffice. 3 ) Rewrote the memdopmap implementation to use a map structure instead of a doubly linked list structure. This avoids linear traversal for each request which depending on the scenario could result in not-insignificant overhead. 4 ) Added the ability to disable buffer acknowledgement via a option using the DCP agent. 5 ) Use a 20MB buffered reader on each memcached connection, the same size as used by KV engine. ... On reverting the change above I see the index progress is quick and back to how it was before we upgraded to gocbcore@9.1.3 . I also grabbed the heap profile upon index build this time around and this is how it looked -   Brett Lawson Charles Dixon  what's the play here? Should FTS .. Wait for a quick fix within gocbcore and an immediate release after Revert aa0eefe within gocbcore and an immediate release after (and perhaps try adding the fixed/corrected change back later) Move FTS back to gocbcore@9.1.2 until a fix is made to this and then skip 9.1.3 altogether. Please be advised that we'll need to unblock QE as quickly as possible.

Brett Lawson added a comment - 18/Mar/21 5:17 PM Hey Abhinav Dangeti , Looking at the code submitted by James Lee , I'm now seeing that there is a slight difference in semantics with regards to the handling of dcp shutdown which is likely leading to this problem. Specifically, the fact that some of the notification channels are now being signalled indiscriminately rather than only under certain circumstances. However, I also think that this might have actually had a bug in it before the changes went in and it simply was less likely to be triggered. I'm going to take a look at this and should have a potential fix (at least if this is indeed the issue) by tomorrow afternoon. P.S. We currently rely heavily on the services using us to test the edges of DCP functionality due to the challenges that exist in testing it. Once I have a potential fix tomorrow, is it going to be possible to rerun the testing and unambiguously validate that it corrected the problem? Cheers, Brett

Abhinav Dangeti added a comment - 18/Mar/21 5:20 PM Sounds good Brett Lawson . Absolutely - I'll be able to quickly test your fix, ping me when you have it up on gerrit.

Keshav Murthy added a comment - 18/Mar/21 8:21 PM Abhinav Dangeti :  QE would like to have a full run shortly.  Will there be any regression/breakage if you revert to gocbcore@9.1.2 now and move to 9.1.3 after verifying+testing the fix?

Abhinav Dangeti added a comment - 18/Mar/21 8:36 PM Keshav Murthy  We've reverted to using gocbcore@9.1.2 while we wait for Brett.

Abhinav Dangeti added a comment - 19/Mar/21 2:49 PM Reducing this to "critical" status for now, as QE isn't blocked by this any longer.

Couchbase Build Team added a comment - 22/Mar/21 1:30 PM Build couchbase-server-7.0.0-4752 contains cbft commit 8de7e56 with commit message: MB-45081 : Upgrade to gocbcore@9.1.3 + fixes

Couchbase Build Team added a comment - 22/Mar/21 1:30 PM Build couchbase-server-7.0.0-4752 contains cbftx commit 18f134c with commit message: MB-45081 : Upgrade to gocbcore@9.1.3 + fixes

Couchbase Build Team added a comment - 22/Mar/21 1:30 PM Build couchbase-server-7.0.0-4752 contains cbgt commit 8e5e597 with commit message: MB-45081 : Upgrade to gocbcore@9.1.3 + fixes

Couchbase Build Team added a comment - 22/Mar/21 1:30 PM Build couchbase-server-7.0.0-4752 contains n1fty commit c4dc5a2 with commit message: MB-45081 : Upgrade to gocbcore@9.1.3 + fixes

Couchbase Build Team added a comment - 22/Mar/21 1:30 PM Build couchbase-server-7.0.0-4752 contains query commit df4fa9a with commit message: MB-45081 : Upgrade to gocbcore@9.1.3 + fixes

Mihir Kamdar added a comment - 01/Apr/21 4:28 PM Closing as verified on the recent cheshire cat builds. Not seeing the failures any more in build sanity