Changes between the 2 builds for XDCR

http://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=6.6.3&fromBuild=9808&toVersion=6.6.5&toBuild=10068&f_analytics-dcp-client=off&f_asterixdb=off&f_backup=off&f_cbas-core=off&f_cbft=off&f_cbgt=off&f_couchbase-cli=off&f_couchdb=off&f_eventing=off&f_go-couchbase=off&f_go_json=off&f_goutils=off&f_goxdcr=on&f_indexing=off&f_kv_engine=off&f_n1fty=off&f_ns_server=off&f_query=off&f_testrunner=off&f_tlm=off&f_voltron=off

This test may have regressed between 6.6.4-9961 to 6.6.5-10068

The changes between the builds include

1. Remove staged transaction XATTR and replicate data, and add a switch to turn off this behavior
2. Enforce TLS
3. A bug fix for compressed binary data
4. Coordinate calls to pools/default endpoint
5. An file descriptor leak fix

I don't see any change that could cause this performance drop.

Looking at each node:

• 105 reached 0 for changes_left at 2022-01-08T16:03:10.405
• 106 reached 0 for changes_left at 2022-01-08T15:59:26.433
• 107 reached 0 for changes_left at 2022-01-08T15:58:32.535
• 108 reached 0 for changes_left at 2022-01-08T15:59:52.491
• 109 reached 0 for changes_left at 2022-01-08T15:59:51.557

Node 105 takes about 4 minutes more than the other nodes to finish replication. That would account for the performance regression.

There is no error or warning in the logs for all nodes. Replication started quickly on all nodes.

In node 105, there is big drop in "rate received from DCP" around last 5 minutes of the test and then recovered (see attachment). That does not happen in other nodes of the same test, neither in the same node of the previous test. However, I am not able to find the reason for this drop yet.

On node 105, resp_wait_time was very high (3X compared with others) at one time. However, I am not able to see anything unusual at target cluster in terms of cpu, memory, disk. While I will continue to look at this, I don't see this caused by any code changes. The changes included in this build wouldn't only affect one node.

Lilei Chen Is compaction running at the target?

In any case, Wayne Siu The regression is within 10% and it may not be due to code defect. Moving it out.

Bo-Chun Wang 

The performance number is much lower in the recent runs of Build 6.6.3-9808

Can you take a look at whether there is any change in the test itself or the test environment?

We didn't make any change to the test. For the test environment, there were two changes.

1. We upgraded python version in perfrunner from 3.6 to 3.9 in Dec. 
2. We upgrade OS version from centos 7.3 to centos 7.9 in Oct.

These are all changes I remembered.

Bo-Chun Wangif the test varies even in the same build, it may not due to code changes. We have seem similar issue with GSI test where there is unexplained perform degradation on the same build after OS version changes.