Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38605

FP to CC - [System Test] : Projector on KV node that is being failed over crashes with error - "feed gen-server crashed: runtime error: index out of range"

    XMLWordPrintable

Details

    Description

      Build : 6.5.1-6288
      Test : -test tests/eventing/mad-hatter/test_eventing_rebalance.yml -scope tests/eventing/mad-hatter/scope_eventing_rebalance.yml
      Scale : 2
      Day : 3rd

      The test has a step to failover 2 KV nodes in the cluster

      [2020-04-02T17:45:12-07:00, sequoiatools/couchbase-cli:6.5:885c38] failover -c 172.23.104.16:8091 --server-failover 172.23.104.18:8091 -u Administrator -p password --force
      		 
      [2020-04-02T17:46:03-07:00, sequoiatools/couchbase-cli:6.5:d04999] failover -c 172.23.104.16:8091 --server-failover 172.23.104.25:8091 -u Administrator -p password --force
      		 
      [2020-04-02T17:46:15-07:00, sequoiatools/couchbase-cli:6.5:a2b707] rebalance -c 172.23.104.16:8091 -u Administrator -p password
      		 
      [2020-04-02T17:58:04-07:00, sequoiatools/cmd:a12734] 60

      During this rebalance operation, the projector process on the incoming KV node - 172.23.104.25 crashed with the following error - 

      2020-04-02T17:46:06.317-07:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_ad0ed13eb12247e5d6aa969e8dfcf8eb(127.0.0.1:8091)] feed gen-server crashed: runtime error: index out of range
      		 
      2020-04-02T17:46:06.317-07:00 [Error]   /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
      		 
      github.com/couchbase/indexing/secondary/projector.(*Feed).genServer.func1(0xc0001b8600)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:477 +0x117
      		 
      panic(0xb85480, 0x16b4210)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.6/go/src/runtime/panic.go:513 +0x1b9
      		 
      github.com/couchbase/indexing/secondary/dcp.Bucket.getMasterNode(...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:221
      		 
      github.com/couchbase/indexing/secondary/dcp.(*Bucket).GetFailoverLogs(0xc005fbcd80, 0xc006390013, 0xc000c40000, 0xcd, 0xcd, 0xc006394180, 0x0, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:59 +0xb43
      		 
      github.com/couchbase/indexing/secondary/projector.(*Feed).bucketDetails(0xc0001b8600, 0xc00be94178, 0x7, 0xc00be94180, 0xb, 0x13, 0xc000c40000, 0xcd, 0xcd, 0x0, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:1583 +0x47b
      		 
      github.com/couchbase/indexing/secondary/projector.(*Feed).bucketFeed(0xc0001b8600, 0xc001000013, 0xc000e26090, 0xd5d180, 0xc00be7d8d0, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:1540 +0x121
      		 
      github.com/couchbase/indexing/secondary/projector.(*Feed).start(0xc0001b8600, 0xc0015fa300, 0x13, 0xc000040191, 0x4)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:900 +0x809
      		 
      github.com/couchbase/indexing/secondary/projector.(*Feed).handleCommand(0xc0001b8600, 0xc0001866c0, 0x4, 0x4, 0xc000158001, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:685 +0x1410
      		 
      github.com/couchbase/indexing/secondary/projector.(*Feed).genServer(0xc0001b8600)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:496 +0x20be
      		 
      created by github.com/couchbase/indexing/secondary/projector.NewFeed
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:127 +0x650

      Not sure if this is a regression as this issue has never been observed in 6.5.0 or 6.5.1 testing and there has been no change to the test.

      Attachments

        Issue Links

          Clones

          Bug - A problem which impairs or prevents the functions of the product. MB-38569 [System Test] : Projector on KV node that is being failed over crashes with error - "feed gen-server crashed: runtime error: index out of range"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-38605
          # Subject Branch Project Status CR V
          124975,2 MB-38605 Avoid projector panic during dataloss scenario unstable indexing Status: MERGED +2 +1

          Activity

            People

              varun.velamuri Varun Velamuri
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty