Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-22897

Indexer crashes when we do start/stop/start indexer rebalance in a loop

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 5.0.0
    • secondary-index
    • None
    •  5.0.0-2025
    • Untriaged
    • Unknown

    Description

      Steps to reproduce:

      1) Create a 3 node cluster : 1 kv, 1 indexer and 1 n1ql

      2) Create multiple secondary indexes. All the indexes are hosted on indexer1 node

      3) Add one more indexer node , lets say indexer2 and start rebalance. Indexes won't be redistributed to indexer2(as per design) after rebalance completes.

      4) Remove indexer1 and start rebalance again.

      5) Stop rebalance.

      6) Repeat steps 4) and 5) in a loop multiple times.

      Rebalance fails with following error messages.

      Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42013cfb0 sp=0xc42013cfa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/net/http/transport.go:1063 +0x50e [goport] 2017/02/20 15:50:58 /opt/couchbase/bin/indexer terminated: signal: aborted

      Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200227b8 sp=0xc4200227b0 created by github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:158 +0x1266 [goport] 2017/02/20 15:50:55 /opt/couchbase/bin/indexer terminated: signal: aborted

       

      Ran curl  -u Administrator:password http://index_node:9102/cleanupRebalance on both the indexer nodes. Fixed the rebalance(However it must be noted, I had to run the above curl commands 3-4 times to get it fixed).

      Logs attached.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            deepkaran.salooja Deepkaran Salooja
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty