Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44431

Panic in indexer : Inconsistent Internal State : Err listen tcp4 :9103: bind: address already in use. Aborted.

    XMLWordPrintable

Details

    • Bug
    • Resolution: User Error
    • Major
    • 7.0.0
    • Cheshire-Cat
    • secondary-index
    • 7.0.0-4476
    • Triaged
    • 1
    • Unknown

    Description

      We see this panic occasionally in perf tests where after fresh install of CB server every time , indexer panics with the following logs 

       IndexList: [ IndexList: [ InstId: 3225932365358724876 Defn: DefnId: 12754694220179827065 Name: myindex Using: memdb Bucket: bucket-1 Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: false NumReplica: 0 InstVersion: 0  SecExprs: <ud>([`alt_email`])</ud>  Desc: [] PartitionScheme: SINGLE  HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false  State: INDEX_STATE_INITIAL RState: RebalActive Stream: INIT_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0}] Async: true SessionId: 1 CollectionId: 0 CollectionAware: true EnableOSO: false RestartTs: <nil> Err listen tcp4 :9103: bind: address already in use. Aborted.panic: Inconsistent Internal State
      goroutine 1 [running]:github.com/couchbase/indexing/secondary/common.CrashOnError(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:413github.com/couchbase/indexing/secondary/indexer.(*indexer).sendStreamUpdateForBuildIndex(0xc00251c380, 0xc000039d78, 0x1, 0x1, 0x3, 0xc0001d37a0, 0x8, 0x209dff0, 0x1, 0x5, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4161 +0x135bgithub.com/couchbase/indexing/secondary/indexer.(*indexer).handleBuildIndex(0xc00251c380, 0x1479780, 0xc00524e7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:2723 +0xeacgithub.com/couchbase/indexing/secondary/indexer.(*indexer).handleAdminMsgs(0xc00251c380, 0x1479780, 0xc00524e7c0, 0x1, 0x2112601) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1389 +0x214github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc00251c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:859 +0x1b6github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc0001ca690, 0x26, 0x7ffc000001ed, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:424 +0x1ef8main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:169 +0x11c4Initializing write barrier = 8000

      The run : http://perf.jenkins.couchbase.com/job/secondary-moi/7342/parameters/ 

      The logs : https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-moi-7342/172.23.97.10.zip 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty