Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: 6.5.0
Affects Version/s: 6.5.0
Component/s: secondary-index
Labels:
- system-test
Environment:
6.5.0-3343

Triage:
Untriaged
Is this a Regression?:
Unknown

Description

I discovered this panic while trying to validate ~~MB-32979~~ in build 6.5.0-3343. Logs attached

Stack trace of the panic (I crashed the indexer quite a few times so there are a lot of stack traces in the logs):

[Fatal] IndexScanSource - panic detected while processing defnId:15598607348245803410, instId:1168639684991923463, index:default/idx3, type:scan, partitions:[0], scans: <ud>([{[1] nil 0 range [{[{1 nil 0}] [1] nil 3 }] <nil>}])</ud>, limit:9223372036854775807, consistency:any_consistency, requestId:fa197b8a-6879-463a-9a74-7e17f94e33da

panic: runtime error: invalid memory address or nil pointer dereference [recovered]

        panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb27e2f]

goroutine 1123 [running]:

panic(0xf40140, 0xc420016070)

        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1

github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine.func1(0xc423858380)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:133 +0xcf

panic(0xf40140, 0xc420016070)

        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243

github.com/couchbase/plasma.(*Reader).NewSnapshotIterator(0x0, 0xc4266917c0, 0x1ad3f60, 0x416f7e, 0xc42536e4b0)

        goproj/src/github.com/couchbase/plasma/plasma.go:1276 +0x2f

github.com/couchbase/indexing/secondary/indexer.(*plasmaSnapshot).Iterate(0xc4275fc0a0, 0x1a6c260, 0xc427e12d80, 0x1a70700, 0xc427e12e00, 0x1a70640, 0x1a59710, 0x0, 0x11a8ea0, 0xc42536e4b0, ...)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2045 +0xb2

github.com/couchbase/indexing/secondary/indexer.(*plasmaSnapshot).Range(0xc4275fc0a0, 0x1a6c260, 0xc427e12d80, 0x1a70700, 0xc427e12e00, 0x1a70640, 0x1a59710, 0x0, 0xc42536e4b0, 0xc422302601, ...)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2030 +0xb2

github.com/couchbase/indexing/secondary/indexer.scanSingleSlice(0xc428c42a00, 0x1a70700, 0xc427e12e00, 0x1a70640, 0x1a59710, 0x0, 0x10b546a, 0x5, 0xc428accc80, 0x1, ...)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:206 +0x41b

github.com/couchbase/indexing/secondary/indexer.scanOne(0xc428c42a00, 0x1a70700, 0xc427e12e00, 0x1a70640, 0x1a59710, 0x0, 0x10b546a, 0x5, 0xc428accc80, 0x1, ...)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:148 +0x11e

github.com/couchbase/indexing/secondary/indexer.scatter(0xc428c42a00, 0x1a70700, 0xc427e12e00, 0x1a70640, 0x1a59710, 0x0, 0x10b546a, 0x5, 0xc428accc80, 0x1, ...)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:55 +0xc6

github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine(0xc423858380, 0x0, 0x0)

        goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:336 +0x810

github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt.func1(0xc42686f6c0, 0xc4202d0f30)

        goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:75 +0x45

created by github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt

        goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:80 +0x66

Steps I used to repro
1.256 MB indexer memory quota

2.Load a bucket with a lot of data (my bucket contains 7M docs exactly with cbworkloadgen)
./cbworkloadgen -n 172.23.104.110:8091 -u Administrator -p password --bucket=default -i 1000000 -s 100 -t 10 -j --prefix=key1_ -r 100

3.Create two indexes on age and name
create index idx on default(name)
create index idx on default(age)

4.Change query timeout to 500 ms (to reproduce the panic quickly make the query timeout 300ms)
curl http://172.23.104.110:8093/admin/settings?internal=ok -d '

{"timeout":500000000}

' -u Administrator:asdsda

5.Change scan timeout to 1 second
curl http://172.23.104.173:9102/settings -u Administrator:asdasda -d '

{"indexer.settings.scan_timeout":10000}

6. run cbc-n1qlback to generate high query load (97-99% cpu usage)
./cbc-n1qlback -u Administrator -P asasdada -U couchbase://172.23.104.110/default -t 1 -f queries2.txt

queries 2.txt is a one line file containing this:

{"statement":"SELECT age FROM default WHERE default.age > 1"}

7. After about ~15-20 minutes you should see the panic, to expedite the panic change the query timeout to 300ms instead of 500ms and you should see the panic every 30 seconds or so.

cbc-n1qlback and cbworkloadgen are couchbase tools located in /opt/couchbase/bin