ep_dcp_replica_items_remaining remains high for hours after loading data on build 7.1.0-1212

In performance runs, after loading data, the runs will wait until all data is persisted. In magma ycsb runs on build 7.1.0-1212, ep_dcp_replica_items_remaining remains high for hours until it hit timeout threshold or it's aborted. The issue is reproducible.

http://perf.jenkins.couchbase.com/job/magma-nvme-ycsb/1199/

2021-08-28T17:11:11 [INFO] Monitoring disk queues: bucket-1

2021-08-28T17:11:11 [INFO] ep_queue_size reached 0

2021-08-28T17:11:11 [INFO] ep_flusher_todo reached 0

2021-08-28T17:11:11 [INFO] ep_diskqueue_items reached 0

2021-08-28T17:11:11 [INFO] vb_active_queue_size reached 0

2021-08-28T17:11:11 [INFO] vb_replica_queue_size reached 0

2021-08-28T17:11:11 [INFO] Monitoring DCP queues: bucket-1

2021-08-28T17:11:11 [INFO] ep_dcp_replica_items_remaining = 131777963

2021-08-28T23:08:26 [INFO] ep_dcp_replica_items_remaining = 131777963

Build timed out (after 480 minutes). Marking the build as aborted.

http://perf.jenkins.couchbase.com/job/magma-nvme-ycsb/1200/

2021-08-29T01:33:14 [INFO] ep_dcp_replica_items_remaining = 89339009

2021-08-29T03:43:26 [INFO] ep_dcp_replica_items_remaining = 53249492

Build was aborted

https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-08-29T033333-ns_1%40172.23.100.135.zip

https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-08-29T033333-ns_1%40172.23.100.136.zip

https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-08-29T033333-ns_1%40172.23.100.137.zip