Uploaded image for project: 'Distributed Transactions Java'
  1. Distributed Transactions Java
  2. TXNJ-99

Lesser GET & SET ops with high CPU usage (at server side) while running with transactions as compared to regular KV loadtest

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not a Bug
    • 1.0.0-beta.1
    • None
    • None

    Description

      Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

      Also high CPU utilization 

      Here is a comparison of two load tests with durability set to None .

       

      stats Transaction Test KV Load Test
      OPS ~11000 ops/sec (7748 trans per sec) ~328880 ops /sec
      cmd_get ~37000 ~164440
      cmd_set ~70000 ~164440
      Throughput  7748 trans/sec ~328880  ops/sec
      server side cpu utilization (%)  ~ 90 % ~90 %
      workload 1 Transaction = 4 READ + 3 UPDATE  1 OPS = 1 READ or 1 UPDATE
      workload Distribution  100% transactions 50:50 READ:UPDATE

       

      Cluster Config :  4 Nodes, 2 Replicas , 12 vCPU, 64 GB RAM

      Test Config : 10M Items , 1KB docSize

      Client Info : YCSB , 1.0.0-beta.1 3.0.0-alpha.6 , Uniform requestdistribution, 480 concurrent workers

      WORKLOADTA : Number of ops in Single Transaction 4 , 4 READS, 3 UPDATE, Durability 0

       

      *Table updated with most recent numbers & test config . 

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          sharath.sulochana Sharath Sulochana (Inactive) created issue -
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Field Original Value New Value
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test* |*KV Load Test*|
          |OPS|80000|175000|
          |cmd_get|30000|87000|
          |cmd_set|50000|87000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|80000|175000|
          |cmd_get|30000|87000|
          |cmd_set|50000|87000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|80000|175000|
          |cmd_get|30000|87000|
          |cmd_set|50000|87000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|80000|175000|
          |cmd_get|~30000|~87000|
          |cmd_set|~50000|~87000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|80000|175000|
          |cmd_get|~30000|~87000|
          |cmd_set|~50000|~87000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~200000|
          |cmd_get|~30000|~100000|
          |cmd_set|~50000|~100000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~200000|
          |cmd_get|~30000|~100000|
          |cmd_set|~50000|~100000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~175000|
          |cmd_get|~30000|~87000|
          |cmd_set|~50000|~187000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~175000|
          |cmd_get|~30000|~87000|
          |cmd_set|~50000|~187000|
          |Throughput |5680 trans/sec|175k ops/sec|
          |cpu utilization (%) |80|35|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~400000|
          |cmd_get|~30000|~200000|
          |cmd_set|~50000|~200000|
          |Throughput |5680 trans/sec|400k ops/sec|
          |cpu utilization (%) |80|38|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Affects Version/s 1.0.0-alpha.5 [ 16172 ]
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Fix Version/s 1.0.0-alpha.5 [ 16172 ]
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Labels mad-hatter performance
          graham.pople Graham Pople made changes -
          Status New [ 10003 ] Open [ 1 ]
          graham.pople Graham Pople made changes -
          Fix Version/s 1.0.0-beta.1 [ 16171 ]
          Fix Version/s 1.0.0-alpha.5 [ 16172 ]
          Affects Version/s 1.0.0-beta.1 [ 16171 ]
          Affects Version/s 1.0.0-alpha.5 [ 16172 ]
          graham.pople Graham Pople made changes -
          Fix Version/s future [ 16170 ]
          Fix Version/s 1.0.0-beta.1 [ 16171 ]
          wayne Wayne Siu made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~400000|
          |cmd_get|~30000|~200000|
          |cmd_set|~50000|~200000|
          |Throughput |5680 trans/sec|400k ops/sec|
          |cpu utilization (%) |80|38|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~400000|
          |cmd_get|~30000|~200000|
          |cmd_set|~50000|~200000|
          |Throughput |5680 trans/sec|400k ops/sec|
          |server side cpu utilization (%) |80|38|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          wayne Wayne Siu made changes -
          Summary Lesser GET & SET ops with high CPU usage while running with transactions as compared to regular KV loadtest Lesser GET & SET ops with high CPU usage (at server side) while running with transactions as compared to regular KV loadtest
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Resolution Not a Bug [ 10200 ]
          Status Open [ 1 ] Closed [ 6 ]
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Actual End 2019-08-08 14:55 (issue has been closed)
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Resolution Not a Bug [ 10200 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~80000|~400000|
          |cmd_get|~30000|~200000|
          |cmd_set|~50000|~200000|
          |Throughput |5680 trans/sec|400k ops/sec|
          |server side cpu utilization (%) |80|38|
          |workload|1 Transaction = 3 READ + 1 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~11000 ops/sec (7748 trans per sec)|~328880 ops /sec|
          |cmd_get|~37000|~164440|
          |cmd_set|~50000|~164440|
          |Throughput |7748 trans/sec|~328880  ops/sec|
          |server side cpu utilization (%) |~ 90 %|~90 %|
          |workload|1 Transaction = 4 READ + 3 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

          *Cluster Config :*  _4 Nodes, 2 Replicas , 12 vCPU, 64 GB RAM_

          *Test Config :* _10M_ Items , _1KB_ docSize

          *Client Info :* _YCSB_ , _1.0.0-beta.1_ _3.0.0-alpha.6_ , _Uniform_ requestdistribution, _480_ concurrent workers

          *WORKLOADTA* : Number of ops in Single Transaction _4_ , _4 READS, 3 UPDATE, Durability 0_

           

          *Table updated with most recent numbers & test config . 

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Description Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~11000 ops/sec (7748 trans per sec)|~328880 ops /sec|
          |cmd_get|~37000|~164440|
          |cmd_set|~50000|~164440|
          |Throughput |7748 trans/sec|~328880  ops/sec|
          |server side cpu utilization (%) |~ 90 %|~90 %|
          |workload|1 Transaction = 4 READ + 3 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

          *Cluster Config :*  _4 Nodes, 2 Replicas , 12 vCPU, 64 GB RAM_

          *Test Config :* _10M_ Items , _1KB_ docSize

          *Client Info :* _YCSB_ , _1.0.0-beta.1_ _3.0.0-alpha.6_ , _Uniform_ requestdistribution, _480_ concurrent workers

          *WORKLOADTA* : Number of ops in Single Transaction _4_ , _4 READS, 3 UPDATE, Durability 0_

           

          *Table updated with most recent numbers & test config . 

           
          Observing lesser number of GET & SET operations while running transactions as compared to regular workloadA load test .

          Also high CPU utilization 

          Here is a comparison of two load tests with durability set to None .

           
          |*stats*|*Transaction Test*|*KV Load Test*|
          |OPS|~11000 ops/sec (7748 trans per sec)|~328880 ops /sec|
          |cmd_get|~37000|~164440|
          |cmd_set|~70000|~164440|
          |Throughput |7748 trans/sec|~328880  ops/sec|
          |server side cpu utilization (%) |~ 90 %|~90 %|
          |workload|1 Transaction = 4 READ + 3 UPDATE| 1 OPS = 1 READ or 1 UPDATE|
          |workload Distribution |100% transactions|50:50 READ:UPDATE|

           

          *Cluster Config :*  _4 Nodes, 2 Replicas , 12 vCPU, 64 GB RAM_

          *Test Config :* _10M_ Items , _1KB_ docSize

          *Client Info :* _YCSB_ , _1.0.0-beta.1_ _3.0.0-alpha.6_ , _Uniform_ requestdistribution, _480_ concurrent workers

          *WORKLOADTA* : Number of ops in Single Transaction _4_ , _4 READS, 3 UPDATE, Durability 0_

           

          *Table updated with most recent numbers & test config . 

           
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Assignee Sharath Sulochana [ sharath.sulochana ] Graham Pople [ graham.pople ]
          graham.pople Graham Pople made changes -
          Attachment image-2019-08-23-14-05-22-430.png [ 72655 ]
          graham.pople Graham Pople made changes -
          Comment [ [~sharath.sulochana] it looks like there may be just one KV endpoint shared between all YCSB threads, which will have a severe impact on throughput especially with durability.  Can you check my working here?
           * Each YCSB worker node runs a single ycsb process.
           * Each process spins up X threads (configurable)
           * Each thread creates a single Couchbase3Client. 
           * Couchbase3Client has some singleton logic so there will only be a single ClusterEnvironment, Cluster, Collection and Transactions object created for the whole ycsb process.  E.g. they're shared between all Couchbase3Clients/threads.
           * The ycsb process takes a couchbase.kvEndpoints param that defaults to 1.  I've looked through some of the perf cluster jobs and as far as I can tell none of them are setting this param.
           * So, we end up with just one KV endpoint for the ycsb process.

          Is that all correct? ]
          graham.pople Graham Pople made changes -
          Fix Version/s .future [ 16170 ]
          Resolution Not a Bug [ 10200 ]
          Status Reopened [ 4 ] Closed [ 6 ]

          People

            graham.pople Graham Pople
            sharath.sulochana Sharath Sulochana (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty