Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.0.0
-
Security Level: Public
-
400-4047
-
Untriaged
-
Unknown
Description
Functional test automation sees this intermittently.
Test to repro:
./testrunner -i /tmp/tuqvm2.ini -t tuqquery.tuq_cluster_ops.QueriesOpsTests.test_failover,nodes_out=2,nodes_init=4,replicas=2,indx_type=GSI,doc-per-day=1
Stack from indexer.log
panic: runtime error: index out of range
|
|
goroutine 12601 [running]:
|
github.com/couchbase/indexing/secondary/dcp.(*Bucket).GetVBmap(0xc20d9ffb00, 0xc20b44ed80, 0x2, 0x4, 0xc20f6b1020, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:156 +0x4ff
|
github.com/couchbase/indexing/secondary/common.addDBSbucket(0x7fff5e7a4af2, 0xe, 0xe1b3b0, 0x7, 0xc20f6b1020, 0x7, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:124 +0x3e3
|
github.com/couchbase/indexing/secondary/common.func·005(0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:208 +0x123
|
github.com/couchbase/indexing/secondary/common.BucketSeqnos(0x7fff5e7a4af2, 0xe, 0xe1b3b0, 0x7, 0xc20f6b1020, 0x7, 0x0, 0x0, 0x0, 0x0, ...)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:214 +0x143
|
github.com/couchbase/indexing/secondary/indexer.func·104(0x1, 0x7f6d8700ac00, 0xc20802afa0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/util.go:84 +0x1da
|
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc20e315910, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x71
|
github.com/couchbase/indexing/secondary/indexer.GetCurrentKVTs(0x7fff5e7a4af2, 0xe, 0xe1b3b0, 0x7, 0xc20f6b1020, 0x7, 0x400, 0x0, 0x0, 0x0, ...)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/util.go:90 +0x269
|
github.com/couchbase/indexing/secondary/indexer.func·101(0x0, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2491 +0x20a
|
...)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/util.go:90 +0x269
|
github.com/couchbase/indexing/secondary/indexer.func·101(0x0, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2491 +0x20a
|
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc20e315c18, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x71
|
github.com/couchbase/indexing/secondary/indexer.func·102()
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2494 +0x9d1
|
created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStats
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2579 +0x225
|
|
goroutine 1 [select]:
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc20803e280)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:397 +0x28c
|
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc20805a630, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:283 +0x26d9
|
main.main()
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:80 +0xa65
|
|
goroutine 5 [syscall, 37 minutes]:
|
os/signal.loop()
|
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
|
created by os/signal.init·1
|
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35
|
|
goroutine 9 [IO wait]:
|
net.(*pollDesc).Wait(0xc208010ed0, 0x72, 0x0, 0x0)
|
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
|
net.(*pollDesc).WaitRead(0xc208010ed0, 0x0, 0x0)
|
...
|
|
|
goroutine 10632 [select]:
|
github.com/couchbase/indexing/secondary/indexer.func·103()
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2644 +0x12c
|
created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).startTimer
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2652 +0x478
|
|
goroutine 12541 [runnable]:
|
net/http.(*persistConn).readLoop(0xc20d0f4630)
|
/usr/local/go/src/net/http/transport.go:928 +0x9ce
|
created by net/http.(*Transport).dialConn
|
/usr/local/go/src/net/http/transport.go:660 +0xc9f
|
|
goroutine 6734 [select, 4 minutes]:
|
github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).run(0xc20c849260)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:160 +0x406
|
created by github.com/couchbase/indexing/secondary/indexer.CreateMutationStreamReader
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:119 +0x87c
|
|
goroutine 12543 [runnable]:
|
github.com/couchbase/indexing/secondary/common.(*vbSeqnosReader).Routine(0xc20bed1e70)
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:70
|
created by github.com/couchbase/indexing/secondary/common.newVbSeqnosReader
|
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:54 +0xb2
|
[goport] 2015/08/18 14:30:38 /opt/couchbase/bin/indexer terminated: exit status 2
|
The indexer process crashed a few times, but was able to restart and index items as expected on the test.
Note: The above test is non-deteministic and may/not always fail.
attaching logs from the cluster jenkins.
Attachments
Issue Links
- relates to
-
MB-17276 Test_failover fails :'msg': 'vbmap: Invalid KV node no -1 for vb 255 from 172.23.107.56:9101
- Closed
For Gerrit Dashboard: MB-16094 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
54572,2 | MB-16094 GetVbMap(): Validate vb map server indexes | unstable | indexing | Status: MERGED | +2 | +1 |
54742,2 | MB-16094: Move SHA to pick up fix on 4.0.0 RC5 | sherlock-4.0.0 | manifest | Status: MERGED | +2 | +1 |