Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49848

Investigate performance gain with TLS=all

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: User Error
    • 7.1.0
    • 7.1.0
    • secondary-index

    Description

      Build: 7.1.0-1745

      Test: 

      1 bucket x 50M x 1KB, 10K KV ops/sec, 1000 indexes, Plasma, s=1 c=100, 128 Concurrency, N2N=all

       

        TLS Non-TLS
      Scan throughput  433518.6 176816.6
      90th Latency 0.79 1.18
      95th Latency 1.15 1.66
      Test jobs  http://perf.jenkins.couchbase.com/job/hemera/3241/console http://perf.jenkins.couchbase.com/job/hemera/3254/console

      Cbmonitor comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hemera_710-1745_apply_scanworkload_cdb8&snapshot=hemera_710-1745_apply_scanworkload_f36d 

      seeing bucket CPU is down from 60% to 20% for TLS

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          varun.velamuri Varun Velamuri added a comment - - edited

          Vikas Chaudhary , Good that you raised this bug. From the test logs, the rows throughput is "0" for the test with TLS=all enabled. This means that none of the indexes are getting scanned.

          From http://perf.jenkins.couchbase.com/job/hemera/3241/consoleFull

          23:06:27 2021-11-26T09:36:27 [INFO] Scan throughput: 433518.60375465493
          23:06:27 2021-11-26T09:36:27 [INFO] Rows throughput: 0.0
          23:06:27 2021-11-26T09:36:27 [INFO] Total Recs in Mem 761154836
          

          From http://perf.jenkins.couchbase.com/job/hemera/3254/console
          **

          21:21:31 2021-11-27T07:51:58 [INFO] Scan throughput: 176816.64002626156
          21:21:58 2021-11-27T07:51:58 [INFO] Rows throughput: 176822.5341107319
          21:21:58 2021-11-27T07:51:58 [INFO] Total Recs in Mem 745375547
          

           

          varun.velamuri Varun Velamuri added a comment - - edited Vikas Chaudhary , Good that you raised this bug. From the test logs, the rows throughput is "0" for the test with TLS=all enabled. This means that none of the indexes are getting scanned. From http://perf.jenkins.couchbase.com/job/hemera/3241/consoleFull 23:06:27 2021-11-26T09:36:27 [INFO] Scan throughput: 433518.60375465493 23:06:27 2021-11-26T09:36:27 [INFO] Rows throughput: 0.0 23:06:27 2021-11-26T09:36:27 [INFO] Total Recs in Mem 761154836 From http://perf.jenkins.couchbase.com/job/hemera/3254/console ** 21:21:31 2021-11-27T07:51:58 [INFO] Scan throughput: 176816.64002626156 21:21:58 2021-11-27T07:51:58 [INFO] Rows throughput: 176822.5341107319 21:21:58 2021-11-27T07:51:58 [INFO] Total Recs in Mem 745375547  
          varun.velamuri Varun Velamuri added a comment - - edited

          From indexer_stats.log, the num_requests stat is "0" for all the indexes on the node. This means that indexer node did not receive the scan request.

          From test logs, cbindexperf started around 2021-11-26T09:34:41

          *23:04:41* 2021-11-26T09:34:41 [INFO] To be applied: ./opt/couchbase/bin/cbindexperf -cluster cen-s705.perf.couchbase.com:8091 -auth="Administrator:password" -configfile tests/gsi/scan_config/config_scan_1000_indexes_1s_100c_1.json -resultfile result.json -statsfile /root/statsfile -cpuprofile cpuprofile.prof -memprofile memprofile.prof
          

          From then on, we can see continuous errors from gometa

          2021-11-26T09:34:43.846-08:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 172.23.100.118:48560)
          2021-11-26T09:34:43.868-08:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.100.118:48562.  Error = tls: first record does not look like a TLS handshake. Kill Pipe.
          2021-11-26T09:34:43.868-08:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
          
          

          varun.velamuri Varun Velamuri added a comment - - edited From indexer_stats.log, the num_requests stat is "0" for all the indexes on the node. This means that indexer node did not receive the scan request. From test logs, cbindexperf started around 2021-11-26T09:34:41 *23:04:41* 2021-11-26T09:34:41 [INFO] To be applied: ./opt/couchbase/bin/cbindexperf -cluster cen-s705.perf.couchbase.com:8091 -auth="Administrator:password" -configfile tests/gsi/scan_config/config_scan_1000_indexes_1s_100c_1.json -resultfile result.json -statsfile /root/statsfile -cpuprofile cpuprofile.prof -memprofile memprofile.prof From then on, we can see continuous errors from gometa 2021-11-26T09:34:43.846-08:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 172.23.100.118:48560) 2021-11-26T09:34:43.868-08:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.100.118:48562. Error = tls: first record does not look like a TLS handshake. Kill Pipe. 2021-11-26T09:34:43.868-08:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate

          Vikas Chaudhary , The cbindexperf command line does not contain enabling TLS for client machine. Hence, the gometa errors. 172.23.100.118 is the client machine. cbindex command line has "-use_tls -cacert ./root.pem" option but not cbindexperf. Please re-run the test with tls enabled for cbindexperf

          Resolving this issue as user error

          varun.velamuri Varun Velamuri added a comment - Vikas Chaudhary , The cbindexperf command line does not contain enabling TLS for client machine. Hence, the gometa errors. 172.23.100.118 is the client machine. cbindex command line has "-use_tls -cacert ./root.pem" option but not cbindexperf. Please re-run the test with tls enabled for cbindexperf Resolving this issue as user error
          vikas.chaudhary Vikas Chaudhary added a comment - fixed as https://review.couchbase.org/c/perfrunner/+/167006  

          People

            vikas.chaudhary Vikas Chaudhary
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty