Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
Morpheus
-
Untriaged
-
1
-
Unknown
Description
We ran partition index tests as part of CBPS-987
We observed the following behaviours as
- Increasing Cores from 16 to 48 will not increase the throughput
| 1 bucket x 100M x 1KB, 2K KV ops/sec, 1 indexe, MOI, s=1 c=1 Cluster : 1 index nodes, 5 KV nodes 3 Partition on meta.id Scan range - 400 documents per scan 5 gsi clients |
max_cpu_percent | scan concurrency | cbindexperf gcpercent | indexer_cpu out of max_cpu_percent | cbindexperf_cpu/ 8000 | Throughput |
| 128 | 200 | 3200 | 5000 | 4800 | 110281.5 | |
| 2400 | 128 | 200 | 1700 | 4900 | 108639.2 | |
| 2000 | 128 | 200 | 1500 | 4800 | 100103.6 | |
| 1600 | 128 | 200 | 1300 | 4600 | 87974.3 |
- Increasing the scan concurrency will not increase the throughput and cbindexperf CPU is not maxing
| max_cpu_percent | scan concurrency | cbindexperf gcpercent | indexer_cpu out of max_cpu_percent | cbindexperf_cpu/ 8000 | Throughput |
| 1600 | 128 | 200 | 1300 | 4700 | 90153.8 |
| 1600 | 256 | 200 | 1300 | 4700 | 93749.1 |
| 1600 | 512 | 200 | 1400 | 4700 | 96854.8 |
| 1600 | 1024 | 200 | 1450 | 4000 | 98312.4 |
- Partition vs non-partion throughput is 2X low
| max_cpu_percent | scan concurrency | cbindexperf gcpercent | indexer_cpu out of max_cpu_percent | cbindexperf_cpu/ 8000 | Throughput | |
| 1600 | 512 | 200 | 1400 | 4700 | 96854.8 | |
| Non-partition | 1600 | 512 | 200 | ~1600 | 6500 | 242650.6 |
- Single node 3 partition vs 3 node 3 partition shows 25% degradation
| max_cpu_percent | scan concurrency | cbindexperf gcpercent | indexer_cpu out of max_cpu_percent | cbindexperf_cpu/ 8000 | Throughput | |
| 1 node 3 partion | 1600 | 512 | 200 | 1400 | 4700 | 96854.8 |
| 3 node 3 partion | 1600 | 512 | 200 | 650 | 5500 | 66836.1 |
Logs and run details are here https://docs.google.com/spreadsheets/d/15h_ykM7PrmcilqIv7uuk1oAuApX_t5ZSkrWSdxqUsw4/edit?usp=sharing