GoXDCR: errors like counter_received_old > counter_received (possible racing condition)

Build

4.0.0-2109

Found during manual testing. Per Xiaomei, this error is a panic, we recovered from but indicates a racing condition in code.

Happened during re-replication to flushed target bucket.

Replication a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1 failed. err=map[xmem_a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1_172.23.105.54:11210_1:counter_received_old=821852, counter_received=821850, dataChan_len_old=25000, dataChan_len=25000] xdcr000 ns_1@172.23.105.44 12:45:24 - Wed May 13, 2015

Replication a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1 started running. xdcr000 ns_1@172.23.105.44 12:39:31 - Wed May 13, 2015
Replication a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1 failed. err=map[xmem_a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1_172.23.105.54:11210_1:counter_received_old=423990, counter_received=423989, dataChan_len_old=25000, dataChan_len=25000] xdcr000 ns_1@172.23.105.44 12:39:18 - Wed May 13, 2015

Replication a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1 started running. xdcr000 ns_1@172.23.105.48 12:35:27 - Wed May 13, 2015
Replication a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1 failed. err=map[xmem_a2922e3fa74609a6594ce4417c4261ce/standardbucket1/standardbucket1_172.23.105.58:11210_0:counter_received_old=303095, counter_received=303095, dataChan_len_old=25000, dataChan_len=25000] xdcr000 ns_1@172.23.105.48 12:35:13 - Wed May 13, 2015