Details
Critical Description
Build 4429
I'm not sure what happened, there was no load on the cluster but I attempted to reproduce an issue and ended up with an unresponsive query service.
The Query UI is returning: Received error 504 when contacting the query service. Try refreshing the browser, as errors may occur if the query service is still warming up or suffering a transient error.
But logs don't show any output from repeated attempts.
The only thing I did see was on one query node (I have 2) is this:
_time=2019-09-30T16:21:29.139+00:00 _level=SEVERE _msg=panic: runtime error: invalid memory address or nil pointer dereference |
_time=2019-09-30T16:21:29.139+00:00 _level=SEVERE _msg=request text: <ud>select keyspaces.name from system:keyspaces;</ud> |
_time=2019-09-30T16:21:29.139+00:00 _level=SEVERE _msg=stack: goroutine 60094 [running]: |
github.com/couchbase/query/execution.(*Context).Recover(0xc0025b8600, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/context.go:633 +0xbc |
panic(0x1cfe700, 0x3211490) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9 |
github.com/couchbase/query/datastore/couchbase.(*namespace).refresh(0xc00049aa20) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:920 +0x29b |
github.com/couchbase/query/datastore/couchbase.(*namespace).KeyspaceNames(0xc00049aa20, 0x7, 0x213f900, 0xc00049aa20, 0x0, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:715 +0x40 |
github.com/couchbase/query/datastore/couchbase.(*namespace).KeyspaceIds(0xc00049aa20, 0x1f657fe, 0x7, 0x213f900, 0xc00049aa20, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:711 +0x2b |
github.com/couchbase/query/datastore/system.(*keyspaceIndex).ScanEntries(0xc0006a4c60, 0xc001f1af30, 0x24, 0x7fffffffffffffff, 0x1f68c67, 0x9, 0x211b180, 0x342fec0, 0xc001ba2b40) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/system/system_keyspace_keyspaces.go:320 +0x173 |
github.com/couchbase/query/execution.(*PrimaryScan).scanEntries(0xc00201c6c0, 0xc0025b8600, 0xc001ba2b40, 0x7fffffffffffffff) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:186 +0x188 |
created by github.com/couchbase/query/execution.(*PrimaryScan).scanPrimary
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:81 +0x1e2 |
|
|
goroutine 60094 [running]: |
github.com/couchbase/query/execution.(*Context).Recover(0xc0025b8600, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/context.go:633 +0xbc |
panic(0x1cfe700, 0x3211490) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9 |
github.com/couchbase/query/datastore/couchbase.(*namespace).refresh(0xc00049aa20) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:920 +0x29b |
github.com/couchbase/query/datastore/couchbase.(*namespace).KeyspaceNames(0xc00049aa20, 0x7, 0x213f900, 0xc00049aa20, 0x0, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:715 +0x40 |
github.com/couchbase/query/datastore/couchbase.(*namespace).KeyspaceIds(0xc00049aa20, 0x1f657fe, 0x7, 0x213f900, 0xc00049aa20, 0x0) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:711 +0x2b |
github.com/couchbase/query/datastore/system.(*keyspaceIndex).ScanEntries(0xc0006a4c60, 0xc001f1af30, 0x24, 0x7fffffffffffffff, 0x1f68c67, 0x9, 0x211b180, 0x342fec0, 0xc001ba2b40) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/system/system_keyspace_keyspaces.go:320 +0x173 |
github.com/couchbase/query/execution.(*PrimaryScan).scanEntries(0xc00201c6c0, 0xc0025b8600, 0xc001ba2b40, 0x7fffffffffffffff) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:186 +0x188 |
created by github.com/couchbase/query/execution.(*PrimaryScan).scanPrimary
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_primary.go:81 +0x1e2 |
But I can't correlate it to the exact time.
Logs uploaded to:
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-100-20-129-90.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-100-20-179-174.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-100-20-88-21.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-34-218-100-219.us-west-2.compute.amazonaws.com.zip (query service)
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-35-163-19-81.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-52-25-2-219.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-54-187-151-36.us-west-2.compute.amazonaws.com.zip (query service
https://s3.amazonaws.com/cb-engineering/perry/queryenginefailure/collectinfo-2019-09-30T163001-ns_1%40ec2-54-203-194-78.us-west-2.compute.amazonaws.com.zip