XDCR panicked in performance tests

Test: Avg. initial XDCR rate (items/sec), 1 -> 1 (2 source nozzles, 4 target nozzles), 1 bucket x 5B x 1KB

Job: http://perf.jenkins.couchbase.com/job/rhea-5node1-10B/49/

Build: 7.1.0-1085

Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-10B-49/172.23.97.21.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-10B-49/172.23.97.22.zip

XDCR panicked during the performance test. The number of replication changes left dropped to 0 from 4.9B.

2021-08-01T05:40:32 [INFO] replication_changes_left = 4,989,889,727

2021-08-01T05:40:34 [INFO] replication_changes_left reached 0

I found this fatal error message in ns_server.goxdcr.log.

fatal error: concurrent map iteration and map write

goroutine 89 [running]:runtime.throw(0xc9dd3f, 0x26)

/home/couchbase/.cbdepscache/exploded/x86_64/go-1.15.8/go/src/runtime/panic.go:1116 +0x72 fp=0xc000f59168 sp=0xc000f59138 pc=0x439e32