Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29763

[FTS] cbft panics under load (scorch)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 5.5.0
    • 5.5.0
    • cbft, fts
    • Cluster: atlas_setupA
      OS: CentOS 7
      CPU: E5-2680 v3 (48 vCPU)
      Memory: 256 GB
      Disk: Samsung PM863

    Description

      Scorch only.

      Works fine when single client thread is running FTS queries. But as soon as concurrency added cbft crashes with "Index out of range".

      Detected in build 5.5.0-2653, list of changes for  2653:

      godeps/src/github.com/blevesearch/bleve
      [+] f83cd37 Merge pull request #916 from sreekanth-cb/dvreader_version_1 [+] a8bd3e6 refactoring the dvReader reuse 
      [+] b22d4fd Merge pull request #922 from abhinavdangeti/master 
      [+] 7c99671 Micro optimizations while estimating mem needed for search [+] 612a791 Merge pull request #921 from steveyen/optimize-scorch-ExternalID 
      [+] d784dda Merge pull request #920 from steveyen/optimize-via-vellum-FTS-Reader-API 
      [+] 23aaeb7 scorch optimize ExternalID() with Segment.DocID() API 
      [+] 0a3f3e4 scorch optimize via vellum.FST.Reader() API 
      [+] 96dad7f dvReader improvement

       

       

      Logs from on of the failed tests:

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-2013/172.23.99.211.zip

       

      The panic:

      2018-05-21T09:25:31.346-07:00 [INFO] cfg_metakv_lean: getCurMetaKvPlanMeta, val: {"path":"/fts/cbgt/cfg/planPIndexesLean/planPIndexesLean-253becfc92c7dec139691c1fa8a15caf-1526919898522/","uuid":"550805f515a7904b","implVersion":"5.5.0"}
      {{ 2018-05-21T09:25:36.347-07:00 [INFO] cfg_metakv_lean: getCurMetaKvPlanMeta, val: {"path":"/fts/cbgt/cfg/planPIndexesLean/planPIndexesLean-253becfc92c7dec139691c1fa8a15caf-1526919898522/","uuid":"550805f515a7904b","implVersion":"5.5.0"}}}
      {{ panic: runtime error: index out of range}}

      ...

      ...

      /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index_impl.go:458 +0x2c2 fp=0xc42ec19e58 sp=0xc42ec19900 pc=0xa01b82
      {{ github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc45fe92270, 0x12631c0, 0xc45fe92390, 0xc4400a2070, 0xc44d411ac0, 0xc444cd8800, 0x0)}}
       /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/cache_bleve.go:110 +0x38a fp=0xc42ec19f18 sp=0xc42ec19e58 pc=0xb3f77a
      {{ github.com/blevesearch/bleve.MultiSearch.func1(0x126d500, 0xc45fe92270, 0xc4400a2070)}}
       /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index_alias_impl.go:458 +0x152 fp=0xc42ec19fc8 sp=0xc42ec19f18 pc=0xa0a9d2
      {{ runtime.goexit()}}
       /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42ec19fd0 sp=0xc42ec19fc8 pc=0x45e8b1
      {{ created by github.com/blevesearch/bleve.MultiSearch}}
       /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index_alias_impl.go:465 +0x19d

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            oleksandr.gyryk Alex Gyryk (Inactive)
            oleksandr.gyryk Alex Gyryk (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty