Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-18594

[FTS] Forestdb assertion "hbmeta.prefix != __null' fails, cbft crashes

    XMLWordPrintable

Details

    • Untriaged
    • No
    • ForestDB: Oct 17 - Nov 4

    Description

      Build
      4.5.0-1720

      To reproduce
      ./testrunner -i INI_FILE.ini -p skip-cleanup=False,get-cbcollect-info=False,get-logs=False,stop-on-failure=False,cluster=D:F,GROUP=ALL -t fts.stable_topology_fts.StableTopFTS.index_query_in_parallel,items=1000000,num_queries=100
      While indexing is in progress, check the UI logs. You will see signs of cbft crash.

      With MOSS, crashes are expensive for cbft-

      The following crash happened after the only kv node in the cluster was swapped with another.

      2016-03-08T17:07:04.274-08:00 [INFO] feed_dcp: OnError, name: default_index_1_695a91964da1a512: bucketName: default, bucketUUID: , err: client.Transmit, err: write tcp 172.23.106.175:37136->172.23.106.139:11210: write: connection reset by peer
      2016-03-08T17:07:04.275-08:00 [INFO] main: meh.OnFeedError, srcType: couchbase, err: client.Transmit, err: write tcp 172.23.106.175:37136->172.23.106.139:11210: write: connection reset by peer
      Assertion in (nil) != 0x7f093dcdbed0 in /home/couchbase/serverjenkins/workspace/watson-unix/forestdb/src/hbtrie.cc:1181
      cbft: /home/couchbase/serverjenkins/workspace/watson-unix/forestdb/src/hbtrie.cc:1181: hbtrie_result _hbtrie_find(hbtrie*, void*, int, void*, list*, uint8_t): Assertion `hbmeta.prefix != __null' failed.
      SIGABRT: abort
      PC=0x7f09698b4625 m=14
      signal arrived during cgo execution

      goroutine 208 [syscall, locked to thread]:
      runtime.cgocall(0xd4f9a0, 0xc82046b988, 0x0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/cgocall.go:120 +0x11b fp=0xc82046b940 sp=0xc82046b910
      github.com/couchbase/goforestdb._Cfunc_fdb_end_transaction(0x7f09641306d0, 0x1130e00, 0x0)
      ??:0 +0x39 fp=0xc82046b988 sp=0xc82046b940
      github.com/couchbase/goforestdb.(*File).EndTransaction(0xc820082748, 0xc8209b1200, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/tx.go:37 +0x19b fp=0xc82046ba10 sp=0xc82046b988
      github.com/couchbase/goforestdb.(*KVStore).ExecuteBatch.func1(0xc82046bb20, 0xc8204ed000, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/kvbatch.go:89 +0x3f fp=0xc82046ba48 sp=0xc82046ba10
      github.com/couchbase/goforestdb.(*KVStore).ExecuteBatch(0xc8204ed000, 0xc827ecf1a0, 0xc82046bc00, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/kvbatch.go:116 +0x46f fp=0xc82046bb08 sp=0xc82046ba48
      github.com/blevesearch/blevex/forestdb.(*Writer).ExecuteBatch(0xc828060230, 0x7f09679c0ec8, 0xc827ecf160, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/blevex/forestdb/writer.go:74 +0x57b fp=0xc82046bcb8 sp=0xc82046bb08
      github.com/blevesearch/bleve/index/store/moss.(*llStore).update(0xc820088150, 0x7f0965b3f298, 0xc826e69680, 0x0, 0x0, 0x0, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/store/moss/lower.go:250 +0x6be fp=0xc82046be70 sp=0xc82046bcb8
      github.com/blevesearch/bleve/index/store/moss.initLowerLevelStore.func1(0x7f0965b3f298, 0xc826e69680, 0x0, 0x0, 0x0, 0x0)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/store/moss/lower.go:64 +0x59 fp=0xc82046beb8 sp=0xc82046be70
      github.com/couchbase/moss.(*collection).runPersister(0xc820231b30)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/persister.go:81 +0x2d4 fp=0xc82046bfa8 sp=0xc82046beb8
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc82046bfb0 sp=0xc82046bfa8
      created by github.com/couchbase/moss.(*collection).Start
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/collection.go:88 +0x63

      goroutine 1 [chan receive, 1 minutes]:
      main.main()
      /home/couchbase/serverjenkins/workspace/watson-unix/goproj/src/github.com/couchbase/cbft/cmd/cbft/main.go:241 +0x217f

      goroutine 17 [syscall, 1 minutes, locked to thread]:
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1

      goroutine 21 [select, 1 minutes, locked to thread]:
      runtime.gopark(0x11d46a0, 0xc82002b728, 0x103a238, 0x6, 0x43b218, 0x2)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/proc.go:185 +0x163
      runtime.selectgoImpl(0xc82002b728, 0x0, 0x18)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/select.go:392 +0xa64
      runtime.selectgo(0xc82002b728)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/select.go:212 +0x12
      runtime.ensureSigM.func1()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/signal1_unix.go:227 +0x353
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1

      goroutine 34 [select]:
      github.com/blevesearch/bleve/index.AnalysisWorker(0xc82010a0c0, 0xc82010a120)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
      created by github.com/blevesearch/bleve/index.NewAnalysisQueue
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2

      goroutine 35 [select]:
      github.com/blevesearch/bleve/index.AnalysisWorker(0xc82010a0c0, 0xc82010a120)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
      created by github.com/blevesearch/bleve/index.NewAnalysisQueue
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2

      goroutine 36 [select]:
      github.com/blevesearch/bleve/index.AnalysisWorker(0xc82010a0c0, 0xc82010a120)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
      created by github.com/blevesearch/bleve/index.NewAnalysisQueue
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2

      goroutine 37 [select]:
      github.com/blevesearch/bleve/index.AnalysisWorker(0xc82010a0c0, 0xc82010a120)
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:70 +0x14d
      created by github.com/blevesearch/bleve/index.NewAnalysisQueue
      /home/couchbase/serverjenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/analysis.go:62 +0xf2

      goroutine 50 [syscall, 1 minutes]:
      os/signal.loop()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/os/signal/signal_unix.go:22 +0x18
      created by os/signal.init.1
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/os/signal/signal_unix.go:28 +0x37

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty