Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59213

Throughput latency regressed for secondary indexes

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      Seeing throughput / latency regression between 7.2.x vs 7.6.0, from 135682 to 87149

       

      {
      		 
          "build": "7.2.2-6316",
      		 
          "value": 135682
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1419",
      		 
          "value": 103169.4
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1483",
      		 
          "value": 94410.4
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1538",
      		 
          "value": 92699.9
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1569",
      		 
          "value": 92186.9
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1607",
      		 
          "value": 90080.3
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1642",
      		 
          "value": 88469
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1656",
      		 
          "value": 87149
      		 
        }

      run: https://perf.jenkins.couchbase.com/job/hemera/9663/ 

      cbmonitor: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hemera_760-1656_apply_scanworkload_6b55&snapshot=hemera_722-6316_apply_scanworkload_e7bc 

      log : https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-9663/cen-s705.perf.couchbase.com.zip  

       

      10k run 

      [
      		 
        {
      		 
          "build": "7.2.2-6316",
      		 
          "value": 38050.5
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1419",
      		 
          "value": 31045.2
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1483",
      		 
          "value": 29247.3
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1538",
      		 
          "value": 30429.6
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1569",
      		 
          "value": 30103.9
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1607",
      		 
          "value": 30394.2
      		 
        },
      		 
        {
      		 
          "build": "7.6.0-1642",
      		 
          "value": 28462.5
      		 
        }
      		 
      ]

      run https://perf.jenkins.couchbase.com/job/hemera/9647/console 

      7.2.3 - https://perf.jenkins.couchbase.com/job/hemera/9704/ 

      cbmonitor :

      https://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hemera_723-6618_apply_scanworkload_4530&snapshot=hemera_760-1642_apply_scanworkload_43fe

      log: 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-9647/cen-s705.perf.couchbase.com.zip

      Attachments

        1. 7_2_3_avg_disk_bps.png
          7_2_3_avg_disk_bps.png
          74 kB
        2. 7_2_3_cpu_utilization.png
          7_2_3_cpu_utilization.png
          85 kB
        3. 7_2_3_memory_stats.png
          7_2_3_memory_stats.png
          91 kB
        4. 7_2_3_memory_used_queue.png
          7_2_3_memory_used_queue.png
          94 kB
        5. 7_2_3_mutation_and_drain_rate.png
          7_2_3_mutation_and_drain_rate.png
          84 kB
        6. 7_2_3_num_requests.png
          7_2_3_num_requests.png
          95 kB
        7. 7_6_0_avg_disk_bps.png
          7_6_0_avg_disk_bps.png
          75 kB
        8. 7_6_0_avg_resident_percent.png
          7_6_0_avg_resident_percent.png
          51 kB
        9. 7_6_0_cpu_utilization.png
          7_6_0_cpu_utilization.png
          86 kB
        10. 7_6_0_memory_stats.png
          7_6_0_memory_stats.png
          93 kB
        11. 7_6_0_memory_used_queue.png
          7_6_0_memory_used_queue.png
          80 kB
        12. 7_6_0_memory_used_queue-1.png
          7_6_0_memory_used_queue-1.png
          80 kB
        13. 7_6_0_mutation_and_drain_rate.png
          7_6_0_mutation_and_drain_rate.png
          86 kB
        14. 7_6_0_num_requests.png
          7_6_0_num_requests.png
          92 kB
        15. same_test_on_MOI.png
          same_test_on_MOI.png
          151 kB
        16. Screenshot 2023-10-19 at 8.00.22 PM.png
          Screenshot 2023-10-19 at 8.00.22 PM.png
          403 kB
        17. Screenshot 2023-10-19 at 8.22.39 PM.png
          Screenshot 2023-10-19 at 8.22.39 PM.png
          368 kB
        18. Screenshot 2024-01-26 at 9.11.32 PM.png
          Screenshot 2024-01-26 at 9.11.32 PM.png
          214 kB
        19. Screenshot 2024-01-26 at 9.11.42 PM.png
          Screenshot 2024-01-26 at 9.11.42 PM.png
          171 kB
        20. Screenshot 2024-01-26 at 9.11.42 PM-1.png
          Screenshot 2024-01-26 at 9.11.42 PM-1.png
          171 kB
        21. Screenshot 2024-01-26 at 9.19.19 PM.png
          Screenshot 2024-01-26 at 9.19.19 PM.png
          216 kB
        22. Screenshot 2024-01-26 at 9.19.29 PM.png
          Screenshot 2024-01-26 at 9.19.29 PM.png
          170 kB
        23. Screenshot 2024-03-21 at 7.12.41 PM.png
          Screenshot 2024-03-21 at 7.12.41 PM.png
          154 kB
        24. Screenshot 2024-03-22 at 10.48.09 AM.png
          Screenshot 2024-03-22 at 10.48.09 AM.png
          178 kB
        25. Screenshot 2024-03-22 at 10.48.22 AM.png
          Screenshot 2024-03-22 at 10.48.22 AM.png
          168 kB

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-60681 [BP-7.2.5] - Throughout latency regressed for 1k and 10k indexes

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          For Gerrit Dashboard: MB-59213
          # Subject Branch Project Status CR V
          204543,1 MB-59213: toy neo nitro Status: NEW 0 0
          204544,4 MB-59213: toy neo plasma Status: NEW 0 0
          204698,2 MB-59213: toy: Comment part of log cleaner change neo plasma Status: NEW 0 0

          Activity

            People

              jinesh.parakh Jinesh Parakh
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are 3 open Gerrit changes

                  PagerDuty