Hitting out of memory issue when loading docs with indexes created

When we run N1QL YCSB tests on aws, we hit out of memory issue when loading docs with indexes created.

Hard out-of-memory error: Bucket "bucket-1" on node XXX is full. No memory currently allocated to this bucket can be easily released.

The AWS instance we use for serve nodes is t3.large. 

OS: CentOS 7, CPU: t3.large (2 vCPU), Memory: 8 GB, Disk: EBS 40GB

In our test configurations, there are 3 server nodes, and we run multiple services (kv/index/n1ql/fts) on each node. The memory quota of each service on a node is listed below.

mem_quota = 256
index_mem_quota = 1024
fts_index_mem_quota = 256

There are 1 bucket and 40 collections. When we tried to load 150K docs with 100 workers, we hit the out-of-memory issue. This issue is reproducible.
Build: 7.0.0-5219
Job: http://perf.jenkins.couchbase.com/job/Cloud-Tester/64/ 
Logs: 
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-Cloud-Tester-64/ec2-34-217-103-182.us-west-2.compute.amazonaws.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-Cloud-Tester-64/ec2-34-218-237-13.us-west-2.compute.amazonaws.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-Cloud-Tester-64/ec2-52-40-243-48.us-west-2.compute.amazonaws.com.zip

When we reduced the number of workers from 100 to 10, we got a good run. It looks like the cluster can't handle loading docs correctly when there are many workers.
Job: http://perf.jenkins.couchbase.com/job/Cloud-Tester/68/
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-Cloud-Tester-68/ec2-18-236-72-97.us-west-2.compute.amazonaws.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-Cloud-Tester-68/ec2-34-215-114-206.us-west-2.compute.amazonaws.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-Cloud-Tester-68/ec2-54-212-38-209.us-west-2.compute.amazonaws.com.zip