Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.1.5
-
Untriaged
-
0
-
Yes
Description
I am seeing initial index build time getting increased upto 115%.Seems like with MB-61565. we are observing this regression.
{}Like in this test.
Initial index (min), 1 bucket x 100M x 1KB, 100K KV ops/sec, single 2i index, varying key(2-1024 char), no mutations during initial build only, PLASMA
172.23.97.3:kv 172.23.97.4:kv 172.23.97.10:index 172.23.97.5:kv 172.23.97.6:kv 172.23.97.7:kv All ubuntu clusters |
mem_quota = 40000 MB |
index_mem_quota = 440000 MB |
indexer.settings.max_cpu_percent = 4800 |
http://showfast.sc.couchbase.com/#/runs/secondary_100M_varying_key_plasma_initial_nyx/7.1.5-3877:
logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8270/172.23.97.10.zip indexer
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8270/172.23.97.3.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8270/172.23.97.4.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8270/172.23.97.5.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8270/172.23.97.6.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8270/172.23.97.7.zip
http://showfast.sc.couchbase.com/#/runs/secondary_100M_varying_key_plasma_initial_nyx/7.1.5-3893
logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8802/172.23.97.10.zip indexer
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8802/172.23.97.3.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8802/172.23.97.4.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8802/172.23.97.5.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8802/172.23.97.6.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-8802/172.23.97.7.zip
Looking into the logs around initial index time in both index nodes ..
at the Periodic Aggregated StorageStats:
7.1.5-3877 vs 7.1.5-3893
Reads are way lesser in MP1 wrt to 7.1.5.
This might be not related to the issue at all but just a hunch.
Looking into the cb monitor for above jobs:
- CPU for whole cluster is looking on lower side with MP1.
- Majority of this lower CPU part seems to be coming from project cpu.
- [bucket-1] [myindex] timings_storage_set for MP1 is much higher.
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=nyx_715-3877_build_secondaryindex_6c35&snapshot=nyx_715-3893_build_secondaryindex_7fbd
Some other jobs where the index build time is affected.
| KPI | Title | 7.1.5-3877 (time in mins) |
7.1.5-3893 (time in mins) |
Delta |
| initial | Initial index (min). 1 bucket x 2B. single 2i index. Initial build only. 50% Resident. PLASMA secondary_2B_initial_only_50res_initial_nyx |
76.5 | 165.1 | 115.8 % |
| initial | Initial index (min). 1 bucket x 500M. single 2i index. Initial build only. 20% Resident. PLASMA secondary_500M_initial_only_20res_initial_nyx |
33.7 | 46.7 | 38.6 % |
| initial | Initial index (min), 1 bucket x 1M x 2KB, 1K KV ops/sec, 100 indexes, MOI secondary_1M_multiple_moi_initial_nyx |
0.9 | 0.8 | 11.1 % |
| initial | Initial index (min). 1 bucket x 100M x 1KB. 100K KV ops/sec. single 2i index. varying key(2-1024 char). no mutations during initial build only. PLASMA secondary_100M_varying_key_plasma_initial_nyx |
3.9 | 7.9 | 102.6 % |
| initial | Initial index (min), 1 bucket x 1M x 2KB, 100 KV ops/sec, 100 indexes, PLASMA secondary_1M_multiple_plasma_initial_nyx |
3.8 | 6.9 | 81.6 % |
and some more.. (hyperlink in the time columns will redirect you to respective jobs)
Attachments
Issue Links
- is caused by
-
-
- Closed
-
Gerrit Reviews
| For Gerrit Dashboard: MB-61608 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 208634,1 | Revert "VULN-2503, bump 1.20 to 1.20.6." | main | golang | Status: NEW | -2 | 0 |
| 208955,1 | MB-61608 log first AllowMarkFirstSnap | 7.1.5 | indexing | Status: NEW | 0 | 0 |