Analytics ingestion rate of 99 datasets on the default collection dropped from 33K to 22K on build 7.0.0-4554

The ingestion rate of 99 datasets on the default collection dropped from 33K to 22K on build 7.0.0-4554. I found that incoming_records_count_total is more than the total number of docs (320M) so it took more time to finish ingestion.

2021-02-28T12:54:28 [INFO] Analytics has 439,109,300 docs (target is 320,102,244)

2021-02-28T12:54:29 [INFO] Number of remaining mutations: 1454543442

2021-02-28T12:54:44 [INFO] Analytics has 439,534,211 docs (target is 320,102,244)

2021-02-28T12:54:44 [INFO] Number of remaining mutations: 1454543442

2021-02-28T12:54:59 [INFO] Analytics has 439,924,812 docs (target is 320,102,244)

2021-02-28T12:54:59 [INFO] Number of remaining mutations: 959381775

2021-02-28T12:55:15 [INFO] Analytics has 440,361,473 docs (target is 320,102,244)

2021-02-28T12:55:15 [INFO] Number of remaining mutations: 959381775

2021-02-28T12:55:30 [INFO] Analytics has 440,813,353 docs (target is 320,102,244)

2021-02-28T12:55:30 [INFO] Number of remaining mutations: 959381775

2021-02-28T12:55:46 [INFO] Analytics has 441,285,768 docs (target is 320,102,244)

2021-02-28T12:55:46 [INFO] Number of remaining mutations: 464216247

2021-02-28T12:56:01 [INFO] Analytics has 441,755,191 docs (target is 320,102,244)

2021-02-28T12:56:01 [INFO] Number of remaining mutations: 464216247

2021-02-28T12:56:16 [INFO] Analytics has 442,153,882 docs (target is 320,102,244)

2021-02-28T12:56:17 [INFO] Number of remaining mutations: 0

Baseline: 7.0.0-4502

Job: http://perf.jenkins.couchbase.com/job/oceanus/5161/ 

Ingestion rate: 33297

Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5161/172.23.96.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5161/172.23.96.57.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5161/172.23.96.5.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5161/172.23.96.7.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5161/172.23.96.8.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5161/172.23.96.9.zip

Compare: 7.0.0-4554

Job: http://perf.jenkins.couchbase.com/job/oceanus/5223/ 

Ingestion rate: 22069

Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5223/172.23.96.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5223/172.23.96.57.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5223/172.23.96.5.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5223/172.23.96.7.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5223/172.23.96.8.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-5223/172.23.96.9.zip