Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21930

For stale=false, throughput test for FDB is timing out & latency shown increase in value

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Observed that 2i performance test for Stale=false throughput for FDB is running forever for 4.7.0-1400 build.
      Here are jobs that ran for 8 hours and then timed out, usually it runs for 4-5 hours - http://perf.jenkins.couchbase.com/job/secondary-fdb/2297/console
      Tried to run cbindexperf tool with small number of repeats when test going on and very low throughput (1-2).

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Jung-Sang Ahn,

            Had a run of both latency and throughput test with latest builds of 4.7.0(1522 & 1550) and observed:

            1. Although latency is not 4x/5x higher, latency is ~50% higher than 4.5.1-2844 and almost double of 4.6.0 builds.
            2. Scan throughput job is getting completed, but throughput is ~20% lower than 4.5.1-2844 and ~30% lower of 4.6.0 builds.
            mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Jung-Sang Ahn , Had a run of both latency and throughput test with latest builds of 4.7.0(1522 & 1550) and observed: Although latency is not 4x/5x higher, latency is ~50% higher than 4.5.1-2844 and almost double of 4.6.0 builds. Scan throughput job is getting completed, but throughput is ~20% lower than 4.5.1-2844 and ~30% lower of 4.6.0 builds.

            Mahesh Mandhare,

            1) There is no change on ForestDB code (that reachable from outside), between build of 4.7.0 1370 and 4.7.0 1550.

            2) Nevertheless, the latency varies from 1100 to 4980.

            3) It implies that

               a) The test is not repeatable, or
               b) It is not caused by ForestDB.

            By the way, If we see a small regression (15~20%), then I think this issue is duplicated to MB-21110, which is caused by C++ refactoring.

            jung-sang Jung-Sang Ahn (Inactive) added a comment - Mahesh Mandhare , 1) There is no change on ForestDB code (that reachable from outside), between build of 4.7.0 1370 and 4.7.0 1550. 2) Nevertheless, the latency varies from 1100 to 4980. 3) It implies that    a) The test is not repeatable, or    b) It is not caused by ForestDB. By the way, If we see a small regression (15~20%), then I think this issue is duplicated to MB-21110 , which is caused by C++ refactoring.

            Hi Mahesh Mandhare

            Same as the case in MB-21110, can you please rerun the test using the latest Spock build? Thanks.

             

            jung-sang Jung-Sang Ahn (Inactive) added a comment - Hi Mahesh Mandhare ,  Same as the case in MB-21110 , can you please rerun the test using the latest Spock build? Thanks.  
            ritam.sharma Ritam Sharma added a comment -

            Mahesh Mandhare - Can you please retest with latest spock build ?

            ritam.sharma Ritam Sharma added a comment - Mahesh Mandhare - Can you please retest with latest spock build ?

            Seems that the regression is fixed in 5.0.0-1700.

            Latency:
            4.5.0-2601: 1011.89
            4.6.0-3572: 740.62
            5.0.0-1700: 558.47

            Throughput:
            4.5.0-2601: 46.6
            4.6.0-3572: 59
            5.0.0-1700: 61.8

            Let me close the ticket. Thanks.

            jung-sang Jung-Sang Ahn (Inactive) added a comment - Seems that the regression is fixed in 5.0.0-1700. Latency: 4.5.0-2601: 1011.89 4.6.0-3572: 740.62 5.0.0-1700: 558.47 Throughput: 4.5.0-2601: 46.6 4.6.0-3572: 59 5.0.0-1700: 61.8 Let me close the ticket. Thanks.

            People

              mahesh.mandhare Mahesh Mandhare (Inactive)
              mahesh.mandhare Mahesh Mandhare (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty