Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build : 7.0.0-3955
Job : 1-node sanity
Test : tuqquery.tuq_index.QueriesViewsTests.test_primary_create_delete_index
Seen this issue only once so far, hence it is an intermittent one.
The test is pretty simple, it drops a primary index and creates it again. After the recreation of the primary index, projector has a crash -
2020-12-07T13:53:30.883-08:00 [Info] VBRT[<-507<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.883-08:00 [Info] VBRT[<-147<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed::reConnectToNodes StartDcpFeed failed for secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/2 with err MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
|
2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/1 failed, trying next
|
2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/3 failed, trying next
|
2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 crashed: runtime error: invalid memory address or nil pointer dereference
|
2020-12-07T13:53:30.884-08:00 [Error] /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
|
github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer.func2(0xc0003d4280, 0xc0004b0011, 0xc00076bf58)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:278 +0x181
|
panic(0xc17920, 0x173e210)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
|
github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).dcpRequestStream(0xc0003d4280, 0x1000110317, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x176b510, 0x1, 0x0, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:479 +0x28b
|
github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc0003d4280, 0xc00637a540, 0x290011)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:302 +0x636
|
created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:181 +0x5cd
|
|
2020-12-07T13:53:30.884-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe(127.0.0.1:8091)] ##11 StartVbStreams("default"): dcp.closed
|
2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-293<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-421<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-549<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-229<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-357<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-613<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
|
Attachments
Issue Links
- backports to
-
MB-44410 [BP MB-43136 6.6.2] - Projector crashed while recreating a primary index after dropping one
-
- Closed
-
The following sequence of events caused this issue:
a. Indexer initiated a mutation topic request to projector for all 1023 vbuckets
b. While, the projector is yet to open DCP streams for all vbuckets (i.e. MTR is still active), the bucket for which this mutation topic request is made is deleted and this resulted in shutting down of DCP connections (From memcached.log)
c. Two DCP connections have been shutdown immediately
d. The other two connections are yet to be closed and projector tried to open a DCP stream from the mutation topic request (As data path is asynchronous to control path, it is valid for projector to process StreamEnd's for some vbuckets while the stream requests are pending for other vbuckets). This DCP stream request resulted in establishing the previously closed connections but that did not succeed as the bucket does not exist anymore
e. As no connection was available, projector ended up with a nil feed that resulted in a panic. An appropriate fix for this would be to handle nil feeds before requesting a stream