Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-1037

SDK3(patch set) kv and query latency ~30% higher than SDK2 with external libcouchbase installed

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1
    • None
    • None
    • 1
    • SDK48: FTS Score/Incl, Docs., SDK51: FLEpt1, Txns Test Compl, SDK2: FLEpt2, TxnsTest, SDK4: 3.1GA,Ruby3.1,TxnTst,FLE

    Description

      Comparing SDK3 to SDK2 kv get/set latency, we see that across all latency percentiles, SDK3 is ~20-30% higher than SDK2. Here are the two tests for comparison:

      SDK 2.5.0 - 24 workers:

      http://perf.jenkins.couchbase.com/job/ares/17051/

      SDK 3 - 24 workers:

      http://perf.jenkins.couchbase.com/job/ares/17049/ 

      In both tests, kv ops are against default scope/collection.

      SDK3 is this patch: git+http://review.couchbase.org/couchbase-python-client@refs/changes/91/135991/3

      Here is a comparison graph:

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_700-3090_access_c015&label=sdk2(24workers)&snapshot=ares_700-3090_access_ae94&label=sdk3(24workers)

      The only difference in these tests, other than sdk version, cur_connections to bucket and ESTABILSHED connections to master node 172.23.133.13. Perhaps these "extra" connections are causing the slow down as the "extra" connections is around 30% more than sdk2 which corresponds to the increase in latency

      Workaround

      Remove any externally installed libcouchbase releases from the system. When run with the built-in libcouchbase, the higher latency is not observed.

      Attachments

        1. profile003.gif
          profile003.gif
          185 kB
        2. sdk2.kv.reads.repeated.perf.data
          31.04 MB
        3. sdk2.perf.data
          34.09 MB
        4. sdk3.kv.reads.repeated.3.perf.data
          36.92 MB
        5. sdk3.perf.data
          32.77 MB
        6. sdk3withlcb293.data
          425 kB
        7. sdk3withlcb306.data
          150 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          korrigan.clark Korrigan Clark (Inactive) created issue -
          ingenthr Matt Ingenthron made changes -
          Field Original Value New Value
          Assignee Ellis Breen [ ellis.breen ] Matt Ingenthron [ ingenthr ]
          Ellis.Breen Ellis Breen made changes -
          Status New [ 10003 ] Open [ 1 ]
          korrigan.clark Korrigan Clark (Inactive) made changes -
          Attachment sdk2.kv.reads.repeated.perf.data [ 109076 ]
          korrigan.clark Korrigan Clark (Inactive) made changes -
          Attachment sdk3.kv.reads.repeated.2.perf.data [ 109077 ]
          korrigan.clark Korrigan Clark (Inactive) made changes -
          Attachment sdk3.kv.reads.repeated.2.perf.data [ 109077 ]
          korrigan.clark Korrigan Clark (Inactive) made changes -
          Attachment sdk3.kv.reads.repeated.3.perf.data [ 109089 ]
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Summary SDK3 kv get/set latency ~30% higher than SDK2 SDK3(patch set) kv get/set latency ~30% higher than SDK2
          ingenthr Matt Ingenthron made changes -
          Summary SDK3(patch set) kv get/set latency ~30% higher than SDK2 SDK3(patch set) kv and query latency ~30% higher than SDK2
          ingenthr Matt Ingenthron made changes -
          Sprint SDK51: …, 2 [ 1308 ]
          ingenthr Matt Ingenthron made changes -
          Rank Ranked lower
          ingenthr Matt Ingenthron made changes -
          Sprint SDK51: …, 2 [ 1308 ] SDK48: FTS Score/Incl, Docs. [ 1307 ]
          ingenthr Matt Ingenthron made changes -
          Rank Ranked lower
          ingenthr Matt Ingenthron made changes -
          Assignee Matt Ingenthron [ ingenthr ] David Kelly [ david.kelly ]
          david.kelly David Kelly made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          david.kelly David Kelly made changes -
          Attachment profile003.gif [ 118168 ]
          ingenthr Matt Ingenthron made changes -
          Sprint SDK48: FTS Score/Incl, Docs. [ 1307 ] SDK48: FTS Score/Incl, Docs., SDK51: FLEpt1, Txns Test Compl [ 1307, 1308 ]
          ingenthr Matt Ingenthron made changes -
          Sprint SDK48: FTS Score/Incl, Docs., SDK51: FLEpt1, Txns Test Compl [ 1307, 1308 ] SDK48: FTS Score/Incl, Docs., SDK51: FLEpt1, Txns Test Compl, SDK2: FLEpt2, TxnsTest [ 1307, 1308, 1309 ]
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Labels performance affects-cc-testing performance
          sharath.sulochana Sharath Sulochana (Inactive) made changes -
          Priority Major [ 3 ] Test Blocker [ 6 ]
          korrigan.clark Korrigan Clark (Inactive) made changes -
          Attachment sdk2.perf.data [ 121859 ]
          Attachment sdk3.perf.data [ 121860 ]
          Attachment sdk3withlcb293.data [ 121861 ]
          Attachment sdk3withlcb306.data [ 121862 ]
          ingenthr Matt Ingenthron made changes -
          Summary SDK3(patch set) kv and query latency ~30% higher than SDK2 SDK3(patch set) kv and query latency ~30% higher than SDK2 with external LCB installed
          ingenthr Matt Ingenthron made changes -
          Description Comparing SDK3 to SDK2 kv get/set latency, we see that across all latency percentiles, SDK3 is ~20-30% higher than SDK2. Here are the two tests for comparison:

          SDK 2.5.0 - 24 workers:

          [http://perf.jenkins.couchbase.com/job/ares/17051/]

          SDK 3 - 24 workers:

          [http://perf.jenkins.couchbase.com/job/ares/17049/

          In both tests, kv ops are against default scope/collection.

          SDK3 is this patch: git+http://review.couchbase.org/couchbase-python-client@refs/changes/91/135991/3

          Here is a comparison graph:

          [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_700-3090_access_c015&label=sdk2(24workers)&snapshot=ares_700-3090_access_ae94&label=sdk3(24workers)]

          The only difference in these tests, other than sdk version, cur_connections to bucket and ESTABILSHED connections to master node 172.23.133.13. Perhaps these "extra" connections are causing the slow down as the "extra" connections is around 30% more than sdk2 which corresponds to the increase in latency
          Comparing SDK3 to SDK2 kv get/set latency, we see that across all latency percentiles, SDK3 is ~20-30% higher than SDK2. Here are the two tests for comparison:

          SDK 2.5.0 - 24 workers:

          [http://perf.jenkins.couchbase.com/job/ares/17051/]

          SDK 3 - 24 workers:

          [http://perf.jenkins.couchbase.com/job/ares/17049/

          In both tests, kv ops are against default scope/collection.

          SDK3 is this patch: git+http://review.couchbase.org/couchbase-python-client@refs/changes/91/135991/3

          Here is a comparison graph:

          [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_700-3090_access_c015&label=sdk2(24workers)&snapshot=ares_700-3090_access_ae94&label=sdk3(24workers)]

          The only difference in these tests, other than sdk version, cur_connections to bucket and ESTABILSHED connections to master node 172.23.133.13. Perhaps these "extra" connections are causing the slow down as the "extra" connections is around 30% more than sdk2 which corresponds to the increase in latency

          h2. Workaround

          Remove any externally installed libcouchbase releases from the system. When run with the built-in libcouchbase, the higher latency is not observed.
          ingenthr Matt Ingenthron made changes -
          Summary SDK3(patch set) kv and query latency ~30% higher than SDK2 with external LCB installed SDK3(patch set) kv and query latency ~30% higher than SDK2 with external libcouchbase installed
          ingenthr Matt Ingenthron made changes -
          Sprint SDK48: FTS Score/Incl, Docs., SDK51: FLEpt1, Txns Test Compl, SDK2: FLEpt2, TxnsTest [ 1307, 1308, 1309 ] SDK48: FTS Score/Incl, Docs., SDK51: FLEpt1, Txns Test Compl, SDK2: FLEpt2, TxnsTest, SDK4: 3.1GA,Ruby3.1,TxnTst,FLE [ 1307, 1308, 1309, 1405 ]
          david.kelly David Kelly made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          wayne Wayne Siu made changes -
          Labels affects-cc-testing performance performance
          ingenthr Matt Ingenthron made changes -
          Priority Test Blocker [ 6 ] Minor [ 4 ]
          ingenthr Matt Ingenthron made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          david.kelly David Kelly made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]

          People

            david.kelly David Kelly
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty