Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
6.6.2-9588 ----> 7.0.0-4979
-
Untriaged
-
Centos 64-bit
-
1
-
No
Description
Steps to Repro
It is an essentially an upgrade of the system test cluster.
1. Start a 6.6.2 system test longevity run.
2. It has following cluster setup
- * 9 data nodes
- * 3 analytics nodes
- * 3 eventing nodes
- * 4 indexing nodes
- * 3 search nodes
- * 3 query nodes
3. It has 10 buckets, fts indexes, analytics datasets, 2i indexes, eventing functions.
4. We do a swap rebalance of 6 node(1 data, 1 index, 1 analytics, 1 fts, 1 query, 1 eventing) which has 6.6.2-9588 with 7.0.0-4979. This woks fine.
5. Failover one fts node 6.6.2-9588 - 172.23.106.207
6. Failover one n1ql node 6.6.2-9588 - 172.23.106.191
7. Now try to graceful failover one 6.6.2-9588 - 172.23.105.90
8. Now I hit into MB-45767.
9. To proceed with the upgrade of the cluster at this point I do multi node hard failover of the following nodes.
172.23.105.90
|
172.23.105.62
|
172.23.105.118
|
172.23.105.25
|
10. Run the following command on all the nodes (172.23.105.90,172.23.105.62,172.23.105.118,172.23.105.25,172.23.106.207,172.23.106.191).
systemctl stop couchbase-server
|
rpm -U http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4979/couchbase-server-enterprise-7.0.0-4979-centos7.x86_64.rpm
|
At this point we saw some panics on the query node.
172.23.106.225
_time=2021-04-19T07:43:15.001-07:00 _level=SEVERE _msg=panic: runtime error: invalid memory address or nil pointer dereference
|
_time=2021-04-19T07:43:15.002-07:00 _level=SEVERE _msg=request text: <ud>select * from system:completed_requests where errorCount > 0 order by requestTime desc</ud>
|
_time=2021-04-19T07:43:15.002-07:00 _level=SEVERE _msg=query context: <ud></ud>
|
_time=2021-04-19T07:43:15.002-07:00 _level=SEVERE _msg=stack: goroutine 222209 [running]:
|
github.com/couchbase/query/server.(*Server).serviceRequest.func1(0x282ef40, 0xc002063000, 0xc000e5a5a0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:900 +0xbb
|
panic(0x22033e0, 0x38b07a0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).checkPoolServices(0xc00026a140, 0x0, 0x0, 0x0, 0x0, 0x27d83c0, 0xc000cee000, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:433 +0x3a
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc00026a140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:400 +0x2e1
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc00026a140, 0x0, 0xc0016b3fc8, 0x15f5bd5, 0x233a5a0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:265 +0x2f
|
github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc000233e90, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:565 +0x1dc
|
github.com/couchbase/query/datastore/system.(*requestLogIndex).State(0xc0000ed090, 0xc0000ed040, 0xc0016b4028, 0x1161d74, 0xc0002e3c68, 0xc03bd28b80, 0x2142c20)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/system/system_keyspace_request_log.go:333 +0xa5
|
github.com/couchbase/query/planner.isValidIndex(0x27d91c0, 0xc0000ed090, 0x4, 0x1)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:583 +0x35
|
github.com/couchbase/query/planner.allIndexes(0x27e3e20, 0xc000cd5940, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4, 0xc0016b4300, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:712 +0x480
|
github.com/couchbase/query/planner.(*builder).buildPredicateScan(0xc0017e4480, 0x27e3e20, 0xc000cd5940, 0xc03fce4280, 0xc034a14b40, 0x27f60a0, 0xc0415acb40, 0x0, 0x0, 0x0, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:213 +0x1ee
|
github.com/couchbase/query/planner.(*builder).buildScan(0xc0017e4480, 0x27e3e20, 0xc000cd5940, 0xc03fce4280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:146 +0x855
|
github.com/couchbase/query/planner.(*builder).selectScan(0xc0017e4480, 0x27e3e20, 0xc000cd5940, 0xc03fce4280, 0x0, 0x0, 0x12, 0xc0016b4700, 0xe2d267)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:48 +0x3a7
|
github.com/couchbase/query/planner.(*builder).VisitKeyspaceTerm(0xc0017e4480, 0xc03fce4280, 0x22184c0, 0x245da20, 0xc0017e4b00, 0x7fe198087368)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select_from.go:204 +0x171
|
github.com/couchbase/query/algebra.(*KeyspaceTerm).Accept(0xc03fce4280, 0x27df120, 0xc0017e4480, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/algebra/from_keyspace.go:69 +0x3b
|
github.com/couchbase/query/planner.(*builder).visitFrom(0xc0017e4480, 0xc03fce4300, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select_from.go:138 +0xab6
|
github.com/couchbase/query/planner.(*builder).VisitSubselect(0xc0017e4480, 0xc03fce4300, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select_sub.go:231 +0xb13
|
github.com/couchbase/query/algebra.(*Subselect).Accept(0xc03fce4300, 0x27df120, 0xc0017e4480, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/algebra/select_sub.go:58 +0x3e
|
github.com/couchbase/query/planner.(*builder).VisitSelect(0xc0017e4480, 0xc043a414a0, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select.go:71 +0x328
|
github.com/couchbase/query/algebra.(*Select).Accept(0xc043a414a0, 0x27eaa40, 0xc0017e4480, 0xc033671830, 0x20300f, 0x13c54bb, 0xc0016b53f0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/algebra/select.go:58 +0x3e
|
github.com/couchbase/query/planner.Build(0x27cdec0, 0xc043a414a0, 0x27e9f00, 0xc00025ea90, 0x27ea260, 0xc000e563c0, 0x24eea5e, 0x7, 0x2440100, 0xc03fce4380, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build.go:31 +0x1a3
|
github.com/couchbase/query/planner.BuildPrepared(0x27cdec0, 0xc043a414a0, 0x27e9f00, 0xc00025ea90, 0x27ea260, 0xc000e563c0, 0x24eea5e, 0x7, 0x100, 0xc03fce4380, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_prepared.go:19 +0xc0
|
github.com/couchbase/query/server.(*Server).getPrepared(0xc000e5a5a0, 0x282ef40, 0xc002063000, 0xc048903180, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:1141 +0x12ae
|
github.com/couchbase/query/server.(*Server).serviceRequest(0xc000e5a5a0, 0x282ef40, 0xc002063000)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:941 +0x16e
|
github.com/couchbase/query/server.(*Server).handleRequest(0xc000e5a5a0, 0x282ef40, 0xc002063000, 0xc000e5a5d0, 0x7)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:622 +0x6c
|
github.com/couchbase/query/server.(*Server).ServiceRequest(0xc000e5a5a0, 0x282ef40, 0xc002063000, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:561 +0x77
|
github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc0008282c0, 0x27acde0, 0xc036c35ce0, 0xc03b7acb00)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:262 +0x35d
|
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000212840, 0x27acde0, 0xc036c35ce0, 0xc03b7ac900)
|
/tmp/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
|
net/http.serverHandler.ServeHTTP(0xc000bba000, 0x27acde0, 0xc036c35ce0, 0xc03b7ac900)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
|
net/http.(*conn).serve(0xc034a14780, 0x27b97a0, 0xc032ae03c0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
|
created by net/http.(*Server).Serve
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
|
|
|
goroutine 222209 [running]:
|
github.com/couchbase/query/server.(*Server).serviceRequest.func1(0x282ef40, 0xc002063000, 0xc000e5a5a0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:900 +0xbb
|
panic(0x22033e0, 0x38b07a0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).checkPoolServices(0xc00026a140, 0x0, 0x0, 0x0, 0x0, 0x27d83c0, 0xc000cee000, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:433 +0x3a
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc00026a140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:400 +0x2e1
|
github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc00026a140, 0x0, 0xc0016b3fc8, 0x15f5bd5, 0x233a5a0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:265 +0x2f
|
github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc000233e90, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:565 +0x1dc
|
github.com/couchbase/query/datastore/system.(*requestLogIndex).State(0xc0000ed090, 0xc0000ed040, 0xc0016b4028, 0x1161d74, 0xc0002e3c68, 0xc03bd28b80, 0x2142c20)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/system/system_keyspace_request_log.go:333 +0xa5
|
github.com/couchbase/query/planner.isValidIndex(0x27d91c0, 0xc0000ed090, 0x4, 0x1)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:583 +0x35
|
github.com/couchbase/query/planner.allIndexes(0x27e3e20, 0xc000cd5940, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4, 0xc0016b4300, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:712 +0x480
|
github.com/couchbase/query/planner.(*builder).buildPredicateScan(0xc0017e4480, 0x27e3e20, 0xc000cd5940, 0xc03fce4280, 0xc034a14b40, 0x27f60a0, 0xc0415acb40, 0x0, 0x0, 0x0, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:213 +0x1ee
|
github.com/couchbase/query/planner.(*builder).buildScan(0xc0017e4480, 0x27e3e20, 0xc000cd5940, 0xc03fce4280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:146 +0x855
|
github.com/couchbase/query/planner.(*builder).selectScan(0xc0017e4480, 0x27e3e20, 0xc000cd5940, 0xc03fce4280, 0x0, 0x0, 0x12, 0xc0016b4700, 0xe2d267)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_scan.go:48 +0x3a7
|
github.com/couchbase/query/planner.(*builder).VisitKeyspaceTerm(0xc0017e4480, 0xc03fce4280, 0x22184c0, 0x245da20, 0xc0017e4b00, 0x7fe198087368)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select_from.go:204 +0x171
|
github.com/couchbase/query/algebra.(*KeyspaceTerm).Accept(0xc03fce4280, 0x27df120, 0xc0017e4480, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/algebra/from_keyspace.go:69 +0x3b
|
github.com/couchbase/query/planner.(*builder).visitFrom(0xc0017e4480, 0xc03fce4300, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select_from.go:138 +0xab6
|
github.com/couchbase/query/planner.(*builder).VisitSubselect(0xc0017e4480, 0xc03fce4300, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select_sub.go:231 +0xb13
|
github.com/couchbase/query/algebra.(*Subselect).Accept(0xc03fce4300, 0x27df120, 0xc0017e4480, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/algebra/select_sub.go:58 +0x3e
|
github.com/couchbase/query/planner.(*builder).VisitSelect(0xc0017e4480, 0xc043a414a0, 0x0, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_select.go:71 +0x328
|
github.com/couchbase/query/algebra.(*Select).Accept(0xc043a414a0, 0x27eaa40, 0xc0017e4480, 0xc033671830, 0x20300f, 0x13c54bb, 0xc0016b53f0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/algebra/select.go:58 +0x3e
|
github.com/couchbase/query/planner.Build(0x27cdec0, 0xc043a414a0, 0x27e9f00, 0xc00025ea90, 0x27ea260, 0xc000e563c0, 0x24eea5e, 0x7, 0x2440100, 0xc03fce4380, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build.go:31 +0x1a3
|
github.com/couchbase/query/planner.BuildPrepared(0x27cdec0, 0xc043a414a0, 0x27e9f00, 0xc00025ea90, 0x27ea260, 0xc000e563c0, 0x24eea5e, 0x7, 0x100, 0xc03fce4380, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/planner/build_prepared.go:19 +0xc0
|
github.com/couchbase/query/server.(*Server).getPrepared(0xc000e5a5a0, 0x282ef40, 0xc002063000, 0xc048903180, 0x0, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:1141 +0x12ae
|
github.com/couchbase/query/server.(*Server).serviceRequest(0xc000e5a5a0, 0x282ef40, 0xc002063000)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:941 +0x16e
|
github.com/couchbase/query/server.(*Server).handleRequest(0xc000e5a5a0, 0x282ef40, 0xc002063000, 0xc000e5a5d0, 0x7)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:622 +0x6c
|
github.com/couchbase/query/server.(*Server).ServiceRequest(0xc000e5a5a0, 0x282ef40, 0xc002063000, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:561 +0x77
|
github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc0008282c0, 0x27acde0, 0xc036c35ce0, 0xc03b7acb00)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:262 +0x35d
|
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000212840, 0x27acde0, 0xc036c35ce0, 0xc03b7ac900)
|
/tmp/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
|
net/http.serverHandler.ServeHTTP(0xc000bba000, 0x27acde0, 0xc036c35ce0, 0xc03b7ac900)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
|
net/http.(*conn).serve(0xc034a14780, 0x27b97a0, 0xc032ae03c0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
|
created by net/http.(*Server).Serve
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
|
Similar kind of panics was also seen on 172.23.106.191. cbcollect_info attached.
Attachments
Issue Links
- relates to
-
CMOS-240 Add a check to see if query nodes are vulnerable to panic during online upgrade with failover + recovery
-
- To Do
-