very high XDCR replication lag over WAN

Replication lag is 100+ sec since build  5.5.0-2935:  http://showfast.sc.couchbase.com/#/timeline/Linux/xdcr/ongoing/all

 

The 2935 was about updating gocb and gocbcore:

http://172.23.120.24/builds/latestbuilds/couchbase-server/vulcan/2935/CHANGELOG

   godeps/src/github.com/couchbase/gocb
   godeps/src/gopkg.in/couchbase/gocbcore.v7

 

Environment:

Cluster: titan_5x5 
OS: CentOS 7 
CPU: E5-2680 v3 (48 vCPU) 
Memory: 256 GB 
Disk: Samsung PM863a 

Test:

95th percentile replication lag
5 -> 5 (2 source nozzles, 4 target nozzles)
1 bucket, 1B x 1KB items,  40K updates/sec ongoing 
WAN 80±4 ms

 

Logs from the run on 5.5.0-2935:
Source:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.105.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.106.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.107.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.108.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.109.zip

Destination:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.100.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.101.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.102.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.103.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.104.zip

 

 

Comparison between 5.5.0-2934 and 2935:

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_550-2935_access_0105&label=5.5.0-2935&snapshot=titan_c1_550-2934_access_16c6&label=5.5.0-2934

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c2_550-2935_access_df73&label=5.5.0-2935&snapshot=titan_c2_550-2934_access_37ae&label=5.5.0-2934