Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: User Error
-
7.1.0
-
Untriaged
-
-
1
-
Unknown
Description
Build: 7.1.0-1745
Test:
1 bucket x 50M x 1KB, 10K KV ops/sec, 1000 indexes, Plasma, s=1 c=100, 128 Concurrency, N2N=all
| TLS | Non-TLS | |
|---|---|---|
| Scan throughput | 433518.6 | 176816.6 |
| 90th Latency | 0.79 | 1.18 |
| 95th Latency | 1.15 | 1.66 |
| Test jobs | http://perf.jenkins.couchbase.com/job/hemera/3241/console | http://perf.jenkins.couchbase.com/job/hemera/3254/console |
Cbmonitor comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hemera_710-1745_apply_scanworkload_cdb8&snapshot=hemera_710-1745_apply_scanworkload_f36d
seeing bucket CPU is down from 60% to 20% for TLS
Attachments
Activity
From indexer_stats.log, the num_requests stat is "0" for all the indexes on the node. This means that indexer node did not receive the scan request.
From test logs, cbindexperf started around 2021-11-26T09:34:41
*23:04:41* 2021-11-26T09:34:41 [INFO] To be applied: ./opt/couchbase/bin/cbindexperf -cluster cen-s705.perf.couchbase.com:8091 -auth="Administrator:password" -configfile tests/gsi/scan_config/config_scan_1000_indexes_1s_100c_1.json -resultfile result.json -statsfile /root/statsfile -cpuprofile cpuprofile.prof -memprofile memprofile.prof
|
From then on, we can see continuous errors from gometa
2021-11-26T09:34:43.846-08:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 172.23.100.118:48560)
|
2021-11-26T09:34:43.868-08:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.100.118:48562. Error = tls: first record does not look like a TLS handshake. Kill Pipe.
|
2021-11-26T09:34:43.868-08:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
|
|
Vikas Chaudhary , The cbindexperf command line does not contain enabling TLS for client machine. Hence, the gometa errors. 172.23.100.118 is the client machine. cbindex command line has "-use_tls -cacert ./root.pem" option but not cbindexperf. Please re-run the test with tls enabled for cbindexperf
Resolving this issue as user error
Vikas Chaudhary , Good that you raised this bug. From the test logs, the rows throughput is "0" for the test with TLS=all enabled. This means that none of the indexes are getting scanned.
From http://perf.jenkins.couchbase.com/job/hemera/3241/consoleFull
23:06:27 2021-11-26T09:36:27 [INFO] Scan throughput: 433518.6037546549323:06:27 2021-11-26T09:36:27 [INFO] Rows throughput: 0.023:06:27 2021-11-26T09:36:27 [INFO] Total Recs in Mem 761154836From http://perf.jenkins.couchbase.com/job/hemera/3254/console
**
21:21:31 2021-11-27T07:51:58 [INFO] Scan throughput: 176816.6400262615621:21:58 2021-11-27T07:51:58 [INFO] Rows throughput: 176822.534110731921:21:58 2021-11-27T07:51:58 [INFO] Total Recs in Mem 745375547