Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44410

[BP MB-43136 6.6.2] - Projector crashed while recreating a primary index after dropping one

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build : 7.0.0-3955
      Job : 1-node sanity
      Test : tuqquery.tuq_index.QueriesViewsTests.test_primary_create_delete_index

      Seen this issue only once so far, hence it is an intermittent one.

      The test is pretty simple, it drops a primary index and creates it again. After the recreation of the primary index, projector has a crash -

      2020-12-07T13:53:30.883-08:00 [Info] VBRT[<-507<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.883-08:00 [Info] VBRT[<-147<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed::reConnectToNodes StartDcpFeed failed for secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/2 with err MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: 
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/1 failed, trying next
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 DcpFeed secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720/3 failed, trying next
      2020-12-07T13:53:30.884-08:00 [Error] DCP[secidx:proj-default-MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe-3314633205471536720] ##11 crashed: runtime error: invalid memory address or nil pointer dereference
      2020-12-07T13:53:30.884-08:00 [Error] 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
      github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer.func2(0xc0003d4280, 0xc0004b0011, 0xc00076bf58)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:278 +0x181
      panic(0xc17920, 0x173e210)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).dcpRequestStream(0xc0003d4280, 0x1000110317, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x176b510, 0x1, 0x0, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:479 +0x28b
      github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc0003d4280, 0xc00637a540, 0x290011)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:302 +0x636
      created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:181 +0x5cd
       
      2020-12-07T13:53:30.884-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe(127.0.0.1:8091)] ##11 StartVbStreams("default"): dcp.closed
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-293<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-421<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-549<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-229<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-357<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      2020-12-07T13:53:30.885-08:00 [Info] VBRT[<-613<-default<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_cad869d7f0696134ba9dab38e7e44dfe] ##11 ##4 StreamEnd
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jeelan.poola Jeelan Poola added a comment -

            Wayne SiuMihir Kamdar This is a crash in projector. Seen on 7.0 but present in 6.6.x as well. Fix is a simple 'nil' check. Request inclusion. Thank you!

            jeelan.poola Jeelan Poola added a comment - Wayne Siu Mihir Kamdar This is a crash in projector. Seen on 7.0 but present in 6.6.x as well. Fix is a simple 'nil' check. Request inclusion. Thank you!

            Wayne Siu we can take this one in 6.6.2

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Wayne Siu we can take this one in 6.6.2

            Build couchbase-server-6.6.2-9522 contains indexing commit 0d16cd0 with commit message:
            MB-44410 Do not addToFeed when all nodeFeeds are nil

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9522 contains indexing commit 0d16cd0 with commit message: MB-44410 Do not addToFeed when all nodeFeeds are nil

            Validated on build 6.6.2-9532. 

             

            Created primary index, dropped it and re-created it again successfully.

            hemant.rajput Hemant Rajput added a comment - Validated on build 6.6.2-9532.    Created primary index, dropped it and re-created it again successfully.

            People

              varun.velamuri Varun Velamuri
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty