Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.0.0
-
None
-
5.0.0-2025
-
Untriaged
-
Unknown
Description
Steps to reproduce:
1) Create a 3 node cluster : 1 kv, 1 indexer and 1 n1ql
2) Create multiple secondary indexes. All the indexes are hosted on indexer1 node
3) Add one more indexer node , lets say indexer2 and start rebalance. Indexes won't be redistributed to indexer2(as per design) after rebalance completes.
4) Remove indexer1 and start rebalance again.
5) Stop rebalance.
6) Repeat steps 4) and 5) in a loop multiple times.
Rebalance fails with following error messages.
Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42013cfb0 sp=0xc42013cfa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/net/http/transport.go:1063 +0x50e [goport] 2017/02/20 15:50:58 /opt/couchbase/bin/indexer terminated: signal: aborted
Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200227b8 sp=0xc4200227b0 created by github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:158 +0x1266 [goport] 2017/02/20 15:50:55 /opt/couchbase/bin/indexer terminated: signal: aborted
Ran curl -u Administrator:password http://index_node:9102/cleanupRebalance on both the indexer nodes. Fixed the rebalance(However it must be noted, I had to run the above curl commands 3-4 times to get it fixed).
Logs attached.