Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29170

System Test : Indexer rebalance out fails after an indexer node crashes

    XMLWordPrintable

Details

    Description

      Build : 5.5.0-2432

      Observed a crash on an indexer node while another indexer node was being rebalanced out. A panic is observed on that indexer node which would have caused the crash:
      panic: Timestamp mismatch between snapshot

      This is the error on diag logs:

      Service 'indexer' exited with status 134. Restarting. Messages:
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSnapshot).Range(0xc4bdcf3270, 0x17ed6e0, 0xc4a1fba800, 0x17f17e0, 0xc4a1fbaa60, 0x17f17e0, 0xc4a1fbaaa0, 0x3, 0xc517ed60c0, 0xc452216e73, ...)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1704 +0xb2 fp=0xc492406e40 sp=0xc492406dd0
      		 
      github.com/couchbase/indexing/secondary/indexer.scanSingleSlice(0xc4c9722d80, 0x17f17e0, 0xc4a1fbaa60, 0x17f17e0, 0xc4a1fbaaa0, 0x3, 0xf53494, 0x5, 0xc5141e4910, 0x1, ...)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:164 +0x41a fp=0xc492406f00 sp=0xc492406e40
      		 
      runtime.goexit()
      		 
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc492406f08 sp=0xc492406f00
      		 
      created by github.com/couchbase/indexing/secondary/indexer.scanMultiple
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:73 +0x583
      		 
      [goport(/opt/couchbase/bin/indexer)] 2018/04/06 23:35:55 child process exited with status 134

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jliang John Liang
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty