Details
-
Bug
-
Resolution: Fixed
-
Major
-
5.5.2, 6.0.0, 6.5.0
-
hebe cluster
OS: CentOS 7
CPU: E5-2680 v3 (48 vCPU)
Memory: 64GB
Disk: Samsung PM863a
-
Untriaged
-
-
Unknown
Description
I'm going through Pavel's legacy so might miss some details or report already known issues.
We have 4-node YCSB Workload E test on partitioned index we run since 5.5.
And today I noticed that it never actually worked. The scan query returns panic error once in a while. Many more such errors under load but even singe query via UI Query Workbench fails occasionally.
Reproduces only on multi node and if there is ORDER BY.
Index:
CREATE PRIMARY INDEX primary_idx ON `bucket-1` PARTITION BY HASH(META().id);
Query:
SELECT RAW meta().id FROM `bucket-1` WHERE meta().id > %1 ORDER BY meta().id LIMIT %2
Error:
_msg=panic: runtime error: invalid memory address or nil pointer dereference
request text:request text:<ud>PREPARE `a271558af22275b2d75217b6de9d952bd64d5841` FROM SELECT RAW meta().id FROM `bucket-1` WHERE meta().id >= $1 ORDER BY meta().id LIMIT $2</ud>
stack:goroutine 563965 [running]:github.com/couchbase/query/execution.(*Context).Recover(0xc42129d500) goproj/src/github.com/couchbase/query/execution/context.go:519 +0xbcpanic(0xea9280, 0x195a130) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/panic.go:489 +0x2cfgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).analyzeOrderBy(0xc423b53a20, 0xc42878c3c0, 0x4, 0x4, 0x8, 0xc4222071e0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:1743 +0x2abgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).scatter(0xc423b53a20, 0xc42988dab8, 0xc4222071e0, 0xc42a8df380, 0x4, 0x4, 0xc426fd2440, 0x4, 0x4, 0xc426fd2460, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:480 +0x521github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).doScan(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc423b53a20, 0x0, 0x0, 0x0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1251 +0xacdgithub.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).Scan3Internal(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc422c4ab90, 0x1, 0x1, 0xc422140000, 0x0, 0x0, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1057 +0x31cgithub.com/couchbase/indexing/secondary/queryport/n1ql.(*secondaryIndex3).Scan3(0xc422618270, 0xc42868b260, 0x24, 0xc422c4ab78, 0x1, 0x1, 0x0, 0x0, 0x0, 0xa, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/n1ql/secondary_index.go:1125 +0x494github.com/couchbase/query/execution.(*IndexScan3).scan(0xc429682000, 0xc42129d500, 0xc42868b3e0, 0x0, 0x0) goproj/src/github.com/couchbase/query/execution/scan_index3.go:173 +0x535created by github.com/couchbase/query/execution.(*IndexScan3).RunOnce.func1 goproj/src/github.com/couchbase/query/execution/scan_index3.go:66 +0x227 goroutine 563965 [running]:github.com/couchbase/query/execution.(*Context).Recover(0xc42129d500) goproj/src/github.com/couchbase/query/execution/context.go:519 +0xbcpanic(0xea9280, 0x195a130) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/panic.go:489 +0x2cfgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).analyzeOrderBy(0xc423b53a20, 0xc42878c3c0, 0x4, 0x4, 0x8, 0xc4222071e0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:1743 +0x2abgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).scatter(0xc423b53a20, 0xc42988dab8, 0xc4222071e0, 0xc42a8df380, 0x4, 0x4, 0xc426fd2440, 0x4, 0x4, 0xc426fd2460, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:480 +0x521github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).doScan(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc423b53a20, 0x0, 0x0, 0x0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1251 +0xacdgithub.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).Scan3Internal(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc422c4ab90, 0x1, 0x1, 0xc422140000, 0x0, 0x0, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1057 +0x31cgithub.com/couchbase/indexing/secondary/queryport/n1ql.(*secondaryIndex3).Scan3(0xc422618270, 0xc42868b260, 0x24, 0xc422c4ab78, 0x1, 0x1, 0x0, 0x0, 0x0, 0xa, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/n1ql/secondary_index.go:1125 +0x494github.com/couchbase/query/execution.(*IndexScan3).scan(0xc429682000, 0xc42129d500, 0xc42868b3e0, 0x0, 0x0) goproj/src/github.com/couchbase/query/execution/scan_index3.go:173 +0x535created by github.com/couchbase/query/execution.(*IndexScan3).RunOnce.func1 goproj/src/github.com/couchbase/query/execution/scan_index3.go:66 +0x227_time=2018-11-05T12:12:52.662-08:00 _level=SEVERE _msg=panic: runtime error: invalid memory address or nil pointer dereference
Logs:
Index. Data, Query on every node
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.207.zip
Attachments
Issue Links
- relates to
-
MB-31900 query panics on ORDER_BY query with partitioned index
- Closed