Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38142

[FTS] Crash caused due to a panic within third-party dependency "RoaringBitmap/roaring"

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Is this a Regression?:
      No

      Description

      Here's the stack trace that crash produces ..

      panic: runtime error: index out of rangepanic: runtime error: index out of range
      goroutine 6008269 [running]:github.com/RoaringBitmap/roaring.(*runIterator16).advanceIfNeeded(0xc159d3b160, 0xc000142519) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/RoaringBitmap/roaring/runcontainer.go:1253 +0x1e5github.com/RoaringBitmap/roaring.(*intIterator).AdvanceIfNeeded(0xc00875f770, 0x12519) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/RoaringBitmap/roaring/roaring.go:263 +0xa3github.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsIterator).nextDocNumAtOrAfter(0xc07b3e0f20, 0x12519, 0x3e800000, 0xc000370800, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/segment/zap/posting.go:665 +0x97github.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsIterator).nextAtOrAfter(0xc07b3e0f20, 0x12519, 0x15, 0x610080, 0xc11910ac80, 0x1b22cf2) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/segment/zap/posting.go:530 +0x4dgithub.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsIterator).Advance(0xc07b3e0f20, 0x12519, 0x14, 0x12519, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/segment/zap/posting.go:525 +0x35github.com/blevesearch/bleve/index/scorch.(*IndexSnapshotTermFieldReader).Advance(0xc003829080, 0xc14d650ed8, 0x8, 0x8, 0xc088ada728, 0xc014a35e48, 0xc014a35e48, 0xc00a2b0a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_index_tfr.go:149 +0x21fgithub.com/blevesearch/bleve/search/searcher.(*TermSearcher).Advance(0xc088ada700, 0xc04e8879e0, 0xc14d650ed8, 0x8, 0x8, 0x8, 0xffffffffffffffff, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/search/searcher/search_term.go:105 +0xe5github.com/blevesearch/bleve/search/searcher.(*DisjunctionHeapSearcher).Advance(0xc00a2b0a00, 0xc04e8879e0, 0xc14d650ed8, 0x8, 0x8, 0x8, 0x8, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/search/searcher/search_disjunction_heap.go:246 +0x1d7github.com/blevesearch/bleve/search/searcher.(*ConjunctionSearcher).advanceChild(0xc239d37b00, 0xc04e8879e0, 0x2, 0xc14d650ed8, 0x8, 0x8, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/search/searcher/search_conjunction.go:251 +0x99github.com/blevesearch/bleve/search/searcher.(*ConjunctionSearcher).Next(0xc239d37b00, 0xc04e8879e0, 0x0, 0xc11910ac80, 0xc014a35db0) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/search/searcher/search_conjunction.go:198 +0x135github.com/blevesearch/bleve/search/collector.(*TopNCollector).Collect(0xc07b666000, 0x1070c20, 0xc03dd5a750, 0x1076e20, 0xc239d37b00, 0x1079ea0, 0xc11910ac80, 0x0, 0x1076e20) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/search/collector/topn.go:213 +0x35agithub.com/blevesearch/bleve.(*indexImpl).SearchInContext(0xc000183490, 0x1070c20, 0xc03dd5a750, 0xc000486420, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index_impl.go:532 +0xe0agithub.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc03dd5a6c0, 0x1070c20, 0xc03dd5a750, 0xc000486420, 0x0, 0xc13815afc0, 0x604f7b) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/cache_bleve.go:110 +0x37fgithub.com/blevesearch/bleve.MultiSearch.func1(0x107b1e0, 0xc03dd5a6c0, 0xc000486420) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index_alias_impl.go:469 +0xf2created by github.com/blevesearch/bleve.MultiSearch /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index_alias_impl.go:476 +0x1e0 

      Related: https://github.com/blevesearch/bleve/issues/1336

      Fix: https://github.com/RoaringBitmap/roaring/pull/229

       Note that this has been fixed for 7.0 builds already .. http://review.couchbase.org/#/c/122552/

       

        Attachments

        For Gerrit Dashboard: MB-38142
        # Subject Branch Project Status CR V

          Activity

          Hide
          abhinav Abhinav Dangeti added a comment -

          Pinging Keshav Murthy for approval to push this for 6.5.1.

          Show
          abhinav Abhinav Dangeti added a comment - Pinging Keshav Murthy  for approval to push this for 6.5.1.
          Hide
          keshav Keshav Murthy added a comment -

          Yes. I approve.  Wayne Siu Please approve as well.

          Show
          keshav Keshav Murthy added a comment - Yes. I approve.  Wayne Siu  Please approve as well.
          Hide
          girish.benakappa Girish Benakappa added a comment -

          Tried the following to reproduce the issue. But couldn't reproduce. As the patch already given to the customer, closing this issue. We can reopen the issue if needed.

          • 280M docs loaded and doc size reduced to 60b
          • Added another search node. So now we have 2 search nodes.
          • A custom index is has been created with fields (both text and number fields)
          • Couple of times, index config has been changed intentionally so to rebuild indexing.
          • Size of the zap files, disk usage and file handles to deleted files are being monitored for every 10 mins.
          • A Boolean FTS query is running for every min.
          • Eagle-eye has been started to look for any errors/panic.

          Same above test has been repeated with 1TB of data.

          Show
          girish.benakappa Girish Benakappa added a comment - Tried the following to reproduce the issue. But couldn't reproduce. As the patch already given to the customer, closing this issue. We can reopen the issue if needed. 280M docs loaded and doc size reduced to 60b Added another search node. So now we have 2 search nodes. A custom index is has been created with fields (both text and number fields) Couple of times, index config has been changed intentionally so to rebuild indexing. Size of the zap files, disk usage and file handles to deleted files are being monitored for every 10 mins. A Boolean FTS query is running for every min. Eagle-eye has been started to look for any errors/panic. Same above test has been repeated with 1TB of data.

            People

            Assignee:
            mihir.kamdar Mihir Kamdar
            Reporter:
            abhinav Abhinav Dangeti
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty