Details
Description
1. create buckets + indexes
2. Load data
Seeing crashes on index service
Service 'indexer' exited with status 1. Restarting. Messages: net/http.(*persistConn).writeLoop(0xc20b711c30)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
[goport] 2015/07/14 19:55:37 /opt/couchbase/bin/indexer terminated: exit status 2
log from above index node is attached.
Part of index logs shows "panic "No valid seq num"
[root@centos-64-x64 logs]# grep -A 20 -B 1 "panic" indexer.log
2015-07-14T19:55:35.927Z-07:00 [Info] DATP[->dataport ":9105"] connection "10.6.2.167:59228" closed !
panic: No Valid Restart Seqno Found
goroutine 147428 [running]:
github.com/couchbase/indexing/secondary/common.CrashOnError(0x7f618b052c00, 0xc20b2d3b20)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:359 +0x50
github.com/couchbase/indexing/secondary/indexer.(*StreamState).adjustNonSnapAlignedVbs(0xc2080c8000, 0xc20e3ca1c0, 0xc20fc90001, 0xc20c5736c8, 0x7)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:555 +0x604
github.com/couchbase/indexing/secondary/indexer.(*StreamState).getRepairTsForBucket(0xc2080c8000, 0x1, 0xc20c5736c8, 0x7, 0xc20cb4c128, 0x7f618b062e68, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:475 +0x16d0
github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream(0xc20805f3e0, 0x1, 0xc20c5736c8, 0x7)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2013 +0xc6
created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStreamConnErrorInternal
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:1089 +0x8cf
goroutine 1 [select]:
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc20804f440)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:396 +0x28c
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc20805c5d0, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:282 +0x26d9
main.main()
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:80 +0xa65
Adding logs from all index + rest of the nodes.
Attachments
For Gerrit Dashboard: MB-15692 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
53290,3 | MB-15692 Store and use Vbuuids in last snap marker | unstable | indexing | Status: MERGED | +2 | +1 |