Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-22897

Indexer crashes when we do start/stop/start indexer rebalance in a loop

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.0.0
    • Fix Version/s: 5.0.0
    • Component/s: secondary-index
    • Labels:
      None
    • Environment:
       5.0.0-2025
    • Triage:
      Untriaged
    • Is this a Regression?:
      Unknown

      Description

      Steps to reproduce:

      1) Create a 3 node cluster : 1 kv, 1 indexer and 1 n1ql

      2) Create multiple secondary indexes. All the indexes are hosted on indexer1 node

      3) Add one more indexer node , lets say indexer2 and start rebalance. Indexes won't be redistributed to indexer2(as per design) after rebalance completes.

      4) Remove indexer1 and start rebalance again.

      5) Stop rebalance.

      6) Repeat steps 4) and 5) in a loop multiple times.

      Rebalance fails with following error messages.

      Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42013cfb0 sp=0xc42013cfa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/net/http/transport.go:1063 +0x50e [goport] 2017/02/20 15:50:58 /opt/couchbase/bin/indexer terminated: signal: aborted

      Service 'indexer' exited with status 1. Restarting. Messages: runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200227b8 sp=0xc4200227b0 created by github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:158 +0x1266 [goport] 2017/02/20 15:50:55 /opt/couchbase/bin/indexer terminated: signal: aborted

       

      Ran curl  -u Administrator:password http://index_node:9102/cleanupRebalance on both the indexer nodes. Fixed the rebalance(However it must be noted, I had to run the above curl commands 3-4 times to get it fixed).

      Logs attached.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          deepkaran.salooja Deepkaran Salooja added a comment -

          http://review.couchbase.org/74074 MB-22897 rebalancer - serialize cleanup paths

          Show
          deepkaran.salooja Deepkaran Salooja added a comment - http://review.couchbase.org/74074 MB-22897 rebalancer - serialize cleanup paths
          Hide
          deepkaran.salooja Deepkaran Salooja added a comment -

          merged to unstable

          Show
          deepkaran.salooja Deepkaran Salooja added a comment - merged to unstable
          Hide
          build-team Couchbase Build Team added a comment -

          Build 5.0.0-2094 contains indexing commit abdb44aa5b5778ddaf7991ecb8558d6b9e11b411 with commit message:
          MB-22897 rebalancer - serialize cleanup paths
          https://github.com/couchbase/indexing/commit/abdb44aa5b5778ddaf7991ecb8558d6b9e11b411

          Show
          build-team Couchbase Build Team added a comment - Build 5.0.0-2094 contains indexing commit abdb44aa5b5778ddaf7991ecb8558d6b9e11b411 with commit message: MB-22897 rebalancer - serialize cleanup paths https://github.com/couchbase/indexing/commit/abdb44aa5b5778ddaf7991ecb8558d6b9e11b411
          Hide
          Balakumaran.Gopal Balakumaran Gopal added a comment -

          Tried doing stop/start reblance on Enterprise Edition 5.0.0 build 2824 15+ times. Faced no crashes.

          Show
          Balakumaran.Gopal Balakumaran Gopal added a comment - Tried doing stop/start reblance on Enterprise Edition 5.0.0 build 2824 15+ times. Faced no crashes.

            People

            • Assignee:
              deepkaran.salooja Deepkaran Salooja
              Reporter:
              Balakumaran.Gopal Balakumaran Gopal
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.