Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51702

Partition index performance not scaling linearly

    XMLWordPrintable

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • Morpheus
    • Morpheus
    • secondary-index
    • Untriaged
    • 1
    • Unknown

    Description

      We ran partition index tests as part of CBPS-987 

      We observed the following behaviours as

      • Increasing Cores from 16 to 48 will not increase the throughput 
      1 bucket x 100M x 1KB, 2K KV ops/sec,
      1 indexe, MOI, s=1 c=1 Cluster : 1 index nodes, 5 KV nodes
      3 Partition on meta.id
      Scan range - 400 documents per scan
      5 gsi clients
      max_cpu_percent scan concurrency cbindexperf gcpercent indexer_cpu out of max_cpu_percent cbindexperf_cpu/ 8000 Throughput
        128 200 3200 5000 4800 110281.5
        2400 128 200 1700 4900 108639.2
        2000 128 200 1500 4800 100103.6
        1600 128 200 1300 4600 87974.3
      • Increasing the scan concurrency will not increase the throughput and cbindexperf CPU is not maxing 
      max_cpu_percent scan concurrency cbindexperf gcpercent indexer_cpu out of max_cpu_percent cbindexperf_cpu/ 8000 Throughput
      1600 128 200 1300 4700 90153.8
      1600 256 200 1300 4700 93749.1
      1600 512 200 1400 4700 96854.8
      1600 1024 200 1450 4000 98312.4
      • Partition vs non-partion throughput is 2X low
        max_cpu_percent scan concurrency cbindexperf gcpercent indexer_cpu out of max_cpu_percent cbindexperf_cpu/ 8000 Throughput
        1600 512 200 1400 4700 96854.8
      Non-partition 1600 512 200 ~1600 6500 242650.6
      • Single node 3 partition vs 3 node 3 partition shows 25% degradation
        max_cpu_percent scan concurrency cbindexperf gcpercent indexer_cpu out of max_cpu_percent cbindexperf_cpu/ 8000 Throughput
      1 node 3 partion 1600 512 200 1400 4700 96854.8
      3 node 3 partion 1600 512 200 650 5500 66836.1

      Logs and run details are here https://docs.google.com/spreadsheets/d/15h_ykM7PrmcilqIv7uuk1oAuApX_t5ZSkrWSdxqUsw4/edit?usp=sharing 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          There are no comments yet on this issue.

          People

            yogendra.acharya Yogendra Acharya (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty