Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.5.0
-
Untriaged
-
-
No
Description
Build
4.5.0-1666
Steps:
1. cluster=D,D+F. One data node, one data + fts node.
2. During index building, hard failover the D+F node.
3. Add back that node and rebalance.
Everything goes on smoothly except for a cbft crash on the D+F node with the following stack trace.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x10 pc=0x52fb65]
goroutine 1691 [running]:
github.com/blevesearch/bleve.(*indexImpl).Batch(0xc8203443f0, 0x0, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index_impl.go:320 +0x105
github.com/couchbase/cbft.(*BleveDestPartition).applyBatchLOCKED.func1(0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:966 +0x85
github.com/couchbase/cbgt.Timer.func1()
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:234 +0x25
github.com/rcrowley/go-metrics.(*StandardTimer).Time(0xc820482c00, 0xc820590a00)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/rcrowley/go-metrics/timer.go:212 +0x3d
github.com/couchbase/cbgt.Timer(0xc821db2110, 0x7ff4e6e528d0, 0xc820482c00, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:235 +0xa7
github.com/couchbase/cbft.(*BleveDestPartition).applyBatchLOCKED(0xc820b57080, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:952 +0xa4
github.com/couchbase/cbft.(*BleveDestPartition).SnapshotStart(0xc820b57080, 0xc8201419c0, 0x3, 0x0, 0x21, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:830 +0x4f
github.com/couchbase/cbgt.(*DestForwarder).SnapshotStart(0xc820179f90, 0xc8201419c0, 0x3, 0x0, 0x21, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/dest_forwarder.go:80 +0xe8
github.com/couchbase/cbgt.(*DCPFeed).SnapshotStart.func1(0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/feed_dcp.go:442 +0x1f4
github.com/couchbase/cbgt.Timer.func1()
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:234 +0x25
github.com/rcrowley/go-metrics.(*StandardTimer).Time(0xc820ad3c80, 0xc8205909c0)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/rcrowley/go-metrics/timer.go:212 +0x3d
github.com/couchbase/cbgt.Timer(0xc8227f5c80, 0x7ff4e6e528d0, 0xc820ad3c80, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/misc.go:235 +0xa7
github.com/couchbase/cbgt.(*DCPFeed).SnapshotStart(0xc820548870, 0x3c1, 0x0, 0x21, 0x2, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbgt/feed_dcp.go:443 +0xd7
github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).handleRecv(0xc8200fc900, 0xc820f5e180, 0xc82121e780, 0xc820f5c000, 0x0, 0x0)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1271 +0x1ff1
github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).worker.func3(0xc820f5e240, 0xc8200fc900, 0xc82121e270, 0xc821692090, 0xc82121e780, 0xc820f5e180, 0x5000000000000)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:986 +0x521
created by github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).worker
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1008 +0x10c1
goroutine 1 [chan receive]:
main.main()
/home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/cmd/cbft/main.go:240 +0x217f
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1
goroutine 43 [select]:
github.com/couchbase/moss.(*collection).runMerger(0xc8202c8180)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/collection.go:331 +0xd8a
created by github.com/couchbase/moss.(*collection).Start
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/collection.go:73 +0x41
goroutine 6 [select]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc820020240, 0xc8200202a0)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2
goroutine 7 [select]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc820020240, 0xc8200202a0)
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
/home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2
Adding links to cbcollect info.