Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43136

Projector crashed while recreating a primary index after dropping one

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build : 7.0.0-3955
      Job : 1-node sanity
      Test : tuqquery.tuq_index.QueriesViewsTests.test_primary_create_delete_index

      Seen this issue only once so far, hence it is an intermittent one.

      The test is pretty simple, it drops a primary index and creates it again. After the recreation of the primary index, projector has a crash - 

      2020-12-07T13:53:30.883-08:00 [Info] VBRT[<-507<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.883-08:00 [Info] VBRT[<-147<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed::reConnectToNodes StartDcpFeed failed for secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/2 with err MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: 
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/1 failed, trying next
      		 
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/3 failed, trying next
      		 
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 crashed: runtime error: invalid memory address or nil pointer dereference
      		 
      2020-12-07T13:53:30.884-08:00 [Error] 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
      		 
      github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer.func2(0xc0003d4280, 0xc0004b0011, 0xc00076bf58)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:278 +0x181
      		 
      panic(0xc17920, 0x173e210)
      		 
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).dcpRequestStream(0xc0003d4280, 0x1000110317, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x176b510, 0x1, 0x0, ...)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:479 +0x28b
      		 
      github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc0003d4280, 0xc00637a540, 0x290011)
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:302 +0x636
      		 
      created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver
      		 
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:181 +0x5cd
      		 
       
      		 
      2020-12-07T13:53:30.884-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe(127.0.0.1:8091)] ##11 StartVbStreams("default"): dcp.closed
      		 
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-293<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-421<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-549<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-229<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-357<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      		 
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-613<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-44410 [BP MB-43136 6.6.2] - Projector crashed while recreating a primary index after dropping one

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-43136
          # Subject Branch Project Status CR V
          141669,2 MB-43136 Do not addToFeed when all nodeFeeds are nil unstable indexing Status: MERGED +2 +1
          146318,2 MB-44410 Do not addToFeed when all nodeFeeds are nil mad-hatter indexing Status: MERGED +2 +1

          Activity

            People

              varun.velamuri Varun Velamuri
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty