Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-18479

[FTS] cbft crashes when the only fts node in cluster is failed over and added back

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.0
    • 4.5.0
    • cbft

    Description

      Build
      4.5.0-1666

      Steps:
      1. cluster=D,D+F. One data node, one data + fts node.
      2. During index building, hard failover the D+F node.
      3. Add back that node and rebalance.
      Everything goes on smoothly except for a cbft crash on the D+F node with the following stack trace.

      panic: runtime error: invalid memory address or nil pointer dereference
      [signal 0xb code=0x1 addr=0x10 pc=0x52fb65]

      goroutine 1691 [running]:
      github.com/blevesearch/bleve.(*indexImpl).Batch(0xc8203443f0, 0x0, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index_impl.go:320 +0x105
      github.com/couchbase/cbft.(*BleveDestPartition).applyBatchLOCKED.func1(0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:966 +0x85
      github.com/couchbase/cbgt.Timer.func1()
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:234 +0x25
      github.com/rcrowley/go-metrics.(*StandardTimer).Time(0xc820482c00, 0xc820590a00)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/rcrowley/go-metrics/timer.go:212 +0x3d
      github.com/couchbase/cbgt.Timer(0xc821db2110, 0x7ff4e6e528d0, 0xc820482c00, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:235 +0xa7
      github.com/couchbase/cbft.(*BleveDestPartition).applyBatchLOCKED(0xc820b57080, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:952 +0xa4
      github.com/couchbase/cbft.(*BleveDestPartition).SnapshotStart(0xc820b57080, 0xc8201419c0, 0x3, 0x0, 0x21, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:830 +0x4f
      github.com/couchbase/cbgt.(*DestForwarder).SnapshotStart(0xc820179f90, 0xc8201419c0, 0x3, 0x0, 0x21, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/dest_forwarder.go:80 +0xe8
      github.com/couchbase/cbgt.(*DCPFeed).SnapshotStart.func1(0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/feed_dcp.go:442 +0x1f4
      github.com/couchbase/cbgt.Timer.func1()
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:234 +0x25
      github.com/rcrowley/go-metrics.(*StandardTimer).Time(0xc820ad3c80, 0xc8205909c0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/rcrowley/go-metrics/timer.go:212 +0x3d
      github.com/couchbase/cbgt.Timer(0xc8227f5c80, 0x7ff4e6e528d0, 0xc820ad3c80, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:235 +0xa7
      github.com/couchbase/cbgt.(*DCPFeed).SnapshotStart(0xc820548870, 0x3c1, 0x0, 0x21, 0x2, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/feed_dcp.go:443 +0xd7
      github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).handleRecv(0xc8200fc900, 0xc820f5e180, 0xc82121e780, 0xc820f5c000, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1271 +0x1ff1
      github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).worker.func3(0xc820f5e240, 0xc8200fc900, 0xc82121e270, 0xc821692090, 0xc82121e780, 0xc820f5e180, 0x5000000000000)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:986 +0x521
      created by github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).worker
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1008 +0x10c1

      goroutine 1 [chan receive]:
      main.main()
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/cmd/cbft/main.go:240 +0x217f

      goroutine 17 [syscall, locked to thread]:
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1

      goroutine 43 [select]:
      github.com/couchbase/moss.(*collection).runMerger(0xc8202c8180)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/collection.go:331 +0xd8a
      created by github.com/couchbase/moss.(*collection).Start
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/collection.go:73 +0x41

      goroutine 6 [select]:
      github.com/blevesearch/bleve/index.AnalysisWorker(0xc820020240, 0xc8200202a0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
      created by github.com/blevesearch/bleve/index.NewAnalysisQueue
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2

      goroutine 7 [select]:
      github.com/blevesearch/bleve/index.AnalysisWorker(0xc820020240, 0xc8200202a0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
      created by github.com/blevesearch/bleve/index.NewAnalysisQueue
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2

      Adding links to cbcollect info.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty