Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51130

py-tpcc performance runs show regression in 7.1.0-2335

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.0
    • 7.1.0
    • query
    • Untriaged
    • 1
    • Yes

    Description

      Py-tpcc results show regression in build 7.1.0-2335 across the board.

      http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/pytpcc/Plasma 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            bo-chun.wang Bo-Chun Wang added a comment -

            I did few runs with build 2369. The performance is improved, but the numbers are still slightly lower than the numbers in build 2284.

            Tpm-c (NewOrders/min), nodes 1, warehouse 100, replica 0, durability majority, scan_consistency request_plus, vCPU 16, s=1 c=9

            Tpm-c (NewOrders/min), nodes 1, warehouse 100, replica 0, durability majority, scan_consistency request_plus, vCPU 16, s=1 c=9, Magma

            bo-chun.wang Bo-Chun Wang added a comment - I did few runs with build 2369. The performance is improved, but the numbers are still slightly lower than the numbers in build 2284. Tpm-c (NewOrders/min), nodes 1, warehouse 100, replica 0, durability majority, scan_consistency request_plus, vCPU 16, s=1 c=9 Build TPM Job 7.1.0-2284 6,481 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/472/ 7.1.0-2335 4,842 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/504/ 7.1.0-2369 6,334 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/522/ Tpm-c (NewOrders/min), nodes 1, warehouse 100, replica 0, durability majority, scan_consistency request_plus, vCPU 16, s=1 c=9, Magma Build TPM Job 7.1.0-2284 6,617 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/459/ 7.1.0-2335 4,912 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/491/ 7.1.0-2369 6,501 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/523/
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

            1 node cluster is not reliable

            One node means less bucket memory. Same amount of documents and less resident.

            Run that use more documents not in memory has high cache miss and  will have low TPM

             

            http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/488/

               7.1.0-2210   TPM: 7146,

            http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/484/

            7.1.0-2210 TPM: 6532
            Each bad run I noticed high ep_cahe_miss_rate may be others also might have difference. This is out of my expertise.

            488 Good run   http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_710-2210_run_tpcc_b015

               ep_cache_miss_rate 0.7%

             

            484 Bad run   http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_710-2210_run_tpcc_da31

               ep_cache_miss_rate 9-10%

             

             

             

             

            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited 1 node cluster is not reliable One node means less bucket memory. Same amount of documents and less resident. Run that use more documents not in memory has high cache miss and  will have low TPM   http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/488/    7.1.0-2210   TPM: 7146, http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/484/ 7.1.0-2210 TPM: 6532 Each bad run I noticed high ep_cahe_miss_rate may be others also might have difference. This is out of my expertise. 488 Good run    http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_710-2210_run_tpcc_b015    ep_cache_miss_rate 0.7%   484 Bad run    http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_710-2210_run_tpcc_da31    ep_cache_miss_rate 9-10%        
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

            make it 1 node reliable change bucket memory 20GB to 35GB and try it

            check 4 node it is back

            http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/pytpcc/Plasma#n1ql_trans_4node_pytpcc_16core_100warehouse_160clients_replica2_dur_maj_scancons_requestplus_hercules_n1ql_trans

             

            FYI: This configuration fix only changes 20MB buffer to 16K buffer. If small difference is acceptable vs huge buffers.

            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited make it 1 node reliable change bucket memory 20GB to 35GB and try it check 4 node it is back http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/pytpcc/Plasma#n1ql_trans_4node_pytpcc_16core_100warehouse_160clients_replica2_dur_maj_scancons_requestplus_hercules_n1ql_trans   FYI: This configuration fix only changes 20MB buffer to 16K buffer. If small difference is acceptable vs huge buffers.
            bo-chun.wang Bo-Chun Wang added a comment -

            I will do few runs with multiple nodes.

            bo-chun.wang Bo-Chun Wang added a comment - I will do few runs with multiple nodes.
            bo-chun.wang Bo-Chun Wang added a comment -

            The results of the 6-node runs show the regression is recovered. I close this ticket.

            Tpm-c (NewOrders/min), nodes 6, warehouse 100, replica 2, durability persistToMajority, scan_consistency request_plus, vCPU 16, s=1 c=9

            Tpm-c (NewOrders/min), nodes 6, warehouse 100, replica 2, durability persistToMajority, scan_consistency request_plus, vCPU 16, s=1 c=9, Magma

            bo-chun.wang Bo-Chun Wang added a comment - The results of the 6-node runs show the regression is recovered. I close this ticket. Tpm-c (NewOrders/min), nodes 6, warehouse 100, replica 2, durability persistToMajority, scan_consistency request_plus, vCPU 16, s=1 c=9 Build TPM Job 7.1.0-2284 9,577 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/478/ 7.1.0-2335 8,555 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/510/ 7.1.0-2369 9,707 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/524/ Tpm-c (NewOrders/min), nodes 6, warehouse 100, replica 2, durability persistToMajority, scan_consistency request_plus, vCPU 16, s=1 c=9, Magma Build TPM Job 7.1.0-2284 12,516 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/465/ 7.1.0-2335 9,612 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/497/ 7.1.0-2369 12,453 http://perf.jenkins.couchbase.com/job/hercules_nest_pytpcc_perfrunner/525/

            People

              bo-chun.wang Bo-Chun Wang
              bo-chun.wang Bo-Chun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty