I am seeing failures with many Eventing workers and 25M+ docs
I create an Eventing function "test_update_2" (attached) with an alias of "bdp_vardata" to a bucket called "crondata" (Memory Quota 7.9GB) and have 64 workers with the following source code:
The source bucket and the bucket that is updated is "crondata" in addition there is a 100MB Eventing meta data bucket "metadata"
I load 25,528,448 document into crondata with a KEYs like todelete01::100006 and data like
once the eventing function runs all documents in bucket "crondata" will be enriched with a new field called "random"
When I run Eventing on my 12 core 2.1Ghz 64 MB Xeon
Linux couch01 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
Couchbase Server 6.6.0-7883 (EE)
Configured with Eventing 256 RAM, Data 7900 MB RAM no other services
The system will process about 7.6 Million doc (mutations) and then I will get LCB_ETMPFAIL errors.
In the UI simply pause then resume every 7 million rows. This proves that Eventing can process the data with 64 workers but something odd is happening where we don't honor some sort of resource constraint.
I believe I also have no issues if I set the workers down form sixty-four (64) to just three (3) workers
I have prepared a video showing exactly how it fails hopefully the video and the uploaded Eventing function will help track down the root cause.