Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
7.0.0-3507
-
Untriaged
-
1
-
Unknown
Description
On running the following test : Secondary index scan latency(ms), 1 bucket x 100M x 1KB, 100K KV ops/sec, 10k indexes, MOI, s=1 c=1000 - seeing the following panic from indexer.
runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
runtime stack:
runtime.throw(0x12317d2, 0x11)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:774 +0x72
runtime.checkmcount()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:600 +0xac
runtime.mcommoninit(0xc535ed9c00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:618 +0x73
runtime.allocm(0xc000074f00, 0x0, 0xc53a7e6e00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1489 +0x150
runtime.newm(0x0, 0xc000074f00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1804 +0x39
runtime.startm(0xc000074f00, 0x207ae00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1969 +0x12a
runtime.handoffp(0xc000074f00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1996 +0x52
runtime.retake(0x56d22fb63618, 0x0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:4444 +0x157
runtime.sysmon()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:4353 +0x120
runtime.mstart1()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1201 +0xc3
runtime.mstart()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1167 +0x6e
Logs are attached :
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-280/cen-s708.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-280/cen-s709.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-280/cen-s710.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-280/cen-s712.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hemera-280/cen-s723.perf.couchbase.com.zip
Job : http://perf.jenkins.couchbase.com/view/GSI/job/hemera/280/console
Cbindexperf Scans are made this way :
{ "Bucket": "bucket-1", "Collection": "collection-1", "High": [ "000028" ], "Id": 1, "Inclusion": 3, "Index": "index1-1", "Limit": 1, "Low": [ "000015" ], "NInterval": 100, "Repeat": 19999, "Scope": "scope-1", "Type": "Range" },
- For every index