Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15692

[system test] indexer exits with error "No valid seq num"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.0.0
    • 4.0.0
    • secondary-index
    • Security Level: Public
    • Untriaged
    • Yes

    Description

      1. create buckets + indexes
      2. Load data

      Seeing crashes on index service
      Service 'indexer' exited with status 1. Restarting. Messages: net/http.(*persistConn).writeLoop(0xc20b711c30)
      /usr/local/go/src/net/http/transport.go:945 +0x41d
      created by net/http.(*Transport).dialConn
      /usr/local/go/src/net/http/transport.go:661 +0xcbc
      [goport] 2015/07/14 19:55:37 /opt/couchbase/bin/indexer terminated: exit status 2

      log from above index node is attached.

      Part of index logs shows "panic "No valid seq num"

      [root@centos-64-x64 logs]# grep -A 20 -B 1 "panic" indexer.log
      2015-07-14T19:55:35.927Z-07:00 [Info] DATP[->dataport ":9105"] connection "10.6.2.167:59228" closed !
      panic: No Valid Restart Seqno Found

      goroutine 147428 [running]:
      github.com/couchbase/indexing/secondary/common.CrashOnError(0x7f618b052c00, 0xc20b2d3b20)
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:359 +0x50
      github.com/couchbase/indexing/secondary/indexer.(*StreamState).adjustNonSnapAlignedVbs(0xc2080c8000, 0xc20e3ca1c0, 0xc20fc90001, 0xc20c5736c8, 0x7)
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:555 +0x604
      github.com/couchbase/indexing/secondary/indexer.(*StreamState).getRepairTsForBucket(0xc2080c8000, 0x1, 0xc20c5736c8, 0x7, 0xc20cb4c128, 0x7f618b062e68, 0x0, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_state.go:475 +0x16d0
      github.com/couchbase/indexing/secondary/indexer.(*timekeeper).repairStream(0xc20805f3e0, 0x1, 0xc20c5736c8, 0x7)
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2013 +0xc6
      created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).handleStreamConnErrorInternal
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:1089 +0x8cf

      goroutine 1 [select]:
      github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc20804f440)
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:396 +0x28c
      github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc20805c5d0, 0x0, 0x0, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:282 +0x26d9
      main.main()
      /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:80 +0xa65

      Adding logs from all index + rest of the nodes.

      Attachments

        For Gerrit Dashboard: MB-15692
        # Subject Branch Project Status CR V

        Activity

          People

            jliang John Liang
            ketaki Ketaki Gangal (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty