Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.5, 6.0.3, 6.5.0
-
6.5.0-4380
-
Untriaged
-
Unknown
Description
Build 6.5.0-4380
Observed following panic in query logs while running high bucket density test.
Note that this panic seen only once. Here are steps I did when we see panic-
- In this case, index+n1ql swap rebalance failed in test.
- I tried manually taking 172.23.97.15 node out and rebalance.
- I stopped rebalance before it completes.
- I made 172.23.97.15 node hard fail over and issued rebalance, this complex successfully. But we see panics in query logs.
Panic-
2019/10/07 02:10:10 http: panic serving 172.23.97.16:55062: runtime error: invalid memory address or nil pointer dereference
|
goroutine 4540 [running]:
|
net/http.(*conn).serve.func1(0xc002d86500)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0
|
panic(0x1cfc080, 0x320d490)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc00012e640, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:304 +0x8db
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc00012e640, 0x0, 0x0, 0x0, 0xc003f52a50)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:263 +0x2f
|
github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc000674b40, 0x2147560, 0xc003f52a60)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:566 +0x1e2
|
github.com/couchbase/query/prepareds.(*preparedCache).getPrepared(0x3427f90, 0x2147560, 0xc003f52a10, 0x7, 0xc003f4eeb8, 0x37, 0x1c70ba0, 0xc003f52a20)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:443 +0x5a1
|
github.com/couchbase/query/prepareds.GetPrepared(0x2147560, 0xc003f52a10, 0xc000000007, 0xc003f4eeb8, 0xc003f529c0, 0x2147560, 0xc003f52a10)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:424 +0x53
|
github.com/couchbase/query/server/http.getPrepared(0x213c100, 0xc003f50f80, 0xc003f4eeb0, 0x8, 0x1c70ba0, 0xc003f529c0, 0xc003f4eeb8, 0xc003ec79d8, 0xca804e, 0xc72591, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:620 +0x12f
|
github.com/couchbase/query/server/http.handlePrepared(0xc0034ad800, 0x213c100, 0xc003f50f80, 0xc003f4eeb0, 0x8, 0x1c70ba0, 0xc003f529c0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:191 +0xa3
|
github.com/couchbase/query/server/http.(*jsonArgs).processParameters(0xc003f50f80, 0xc0034ad800, 0xd, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:1170 +0x158
|
github.com/couchbase/query/server/http.newHttpRequest(0xc0034ad800, 0x2129760, 0xc002923500, 0xc00316d200, 0x21281e0, 0xc001e68000, 0x4000000, 0xc000406360)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:116 +0x778
|
github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc001e9a000, 0x2129760, 0xc002923500, 0xc00316d200)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:153 +0x80
|
github.com/gorilla/mux.(*Router).ServeHTTP(0xc001e66050, 0x2129760, 0xc002923500, 0xc00316d200)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/gorilla/mux/mux.go:114 +0xe0
|
net/http.serverHandler.ServeHTTP(0xc001ba9ee0, 0x2129760, 0xc002923500, 0xc00316d000)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2741 +0xab
|
net/http.(*conn).serve(0xc002d86500, 0x212f1e0, 0xc003607e80)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1847 +0x646
|
created by net/http.(*Server).Serve
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2851 +0x2f5
|
...
|
Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/320
Logs-
Index+n1ql nodes-
Panic in logs- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.15.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.19.zip
Panic in logs-https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.20.zip
Other nodes-
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.16.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.17.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.20.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.23.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.14.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.177.zip