Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44216

FTS: panic found during fts query execution

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • fts

    Description

      Build: 7.0.0-4390

      Panic found in server logs during regression tests analysis:

      Trying to execute the following fts query:

      {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "term": "Sofiya", "fuzziness": 2}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}, "highlight": {"style": "ansi"}}
      

      Corresponding test execution log fragment:

      [2021-02-08 12:33:58,957] - [fts_base:2628] INFO - Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "term": "Sofiya", "fuzziness": 2}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}, "highlight": {"style": "ansi"}} on node: 172.23.123.51:
      [2021-02-08 12:33:59,069] - [stable_topology_fts:1974] INFO - Hits: 0
      [2021-02-08 12:33:59,069] - [stable_topology_fts:1975] INFO - Content: []
      [2021-02-08 12:33:59,071] - [remote_util:298] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5
      [2021-02-08 12:33:59,171] - [remote_util:336] INFO - SSH Connected to 172.23.123.51 as root
      [2021-02-08 12:33:59,410] - [remote_util:3555] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True
      [2021-02-08 12:33:59,703] - [remote_util:3711] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
      [2021-02-08 12:33:59,704] - [remote_util:298] INFO - SSH Connecting to 172.23.123.53 with username:root, attempt#1 of 5
      [2021-02-08 12:33:59,805] - [remote_util:336] INFO - SSH Connected to 172.23.123.53 as root
      [2021-02-08 12:34:00,072] - [remote_util:3555] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True
      [2021-02-08 12:34:00,371] - [remote_util:3711] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
      [2021-02-08 12:34:00,372] - [remote_util:298] INFO - SSH Connecting to 172.23.123.52 with username:root, attempt#1 of 5
      [2021-02-08 12:34:00,469] - [remote_util:336] INFO - SSH Connected to 172.23.123.52 as root
      [2021-02-08 12:34:00,720] - [remote_util:3555] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True
      [2021-02-08 12:34:01,011] - [remote_util:3711] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
      Number of matches : 3
      matches : ['2021/02/08 12:33:48 http: panic serving 127.0.0.1:36252: runtime error: invalid memory address or nil pointer dereference', 'panic(0xffeba0, 0x1cb22b0)', '\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2']
      

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          abhinav Abhinav Dangeti added a comment - - edited

          Here's the full stack trace ..

          2021/02/08 12:33:48 http: panic serving 127.0.0.1:36252: runtime error: invalid memory address or nil pointer dereference
          goroutine 1069 [running]:
          net/http.(*conn).serve.func1(0xc0004301e0)
          	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1767 +0x139
          panic(0xffeba0, 0x1cb22b0)
          	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
          github.com/couchbase/moss.(*Store).Histograms(...)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/moss@v0.1.0/store_stats.go:89
          github.com/blevesearch/bleve/v2/index/upsidedown/store/moss.(*stats).statsMap(0xc0000110c0, 0x7faa69bc5a70)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.1/index/upsidedown/store/moss/stats.go:48 +0x27b
          github.com/blevesearch/bleve/v2/index/upsidedown/store/moss.(*Store).StatsMap(0xc0020f2820, 0x13431c0)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.1/index/upsidedown/store/moss/store.go:214 +0x2f
          github.com/blevesearch/bleve/v2/index/upsidedown.(*indexStat).statsMap(0xc0020f2780, 0xc0040ba780)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.1/index/upsidedown/stats.go:46 +0x478
          github.com/blevesearch/bleve/v2/index/upsidedown.(*UpsideDownCouch).StatsMap(0xc0020f0480, 0x6)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.1/index/upsidedown/upsidedown.go:1022 +0x2f
          github.com/blevesearch/bleve/v2.(*IndexStat).statsMap(0xc0020efb40, 0xc0040ba750)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.1/index_stats.go:31 +0x4a
          github.com/blevesearch/bleve/v2.(*indexImpl).StatsMap(0xc0002c8690, 0x0)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.1/index_impl.go:791 +0x2f
          github.com/couchbase/cbft.(*BleveDest).StatsMap(0xc0020ddad0, 0xf930e0, 0xc003eba100, 0xc00225e270)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/pindex_bleve.go:1911 +0xb2
          github.com/couchbase/cbft.addPIndexStats(0xc001cd1300, 0xc003ed9b00, 0x11a15dd, 0x13)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/ns_server.go:596 +0x73
          github.com/couchbase/cbft.gatherIndexStats(0xc000422e00, 0xc001fdf000, 0x1, 0xc0003d5a40, 0x1c, 0xc0044c5460)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/ns_server.go:350 +0x165a
          github.com/couchbase/cbft.(*PrometheusHighMetricsHandler).ServeHTTP(0xc0003e7fe0, 0x1343c40, 0xc003aa0760, 0xc0039dad00)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/prometheus.go:113 +0x144
          github.com/couchbase/cbgt/rest.(*HandlerWithRESTMeta).ServeHTTP(0xc00050e480, 0x13455c0, 0xc000518ee0, 0xc0039dad00)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/rest/rest.go:254 +0x10c
          github.com/couchbase/cbft.(*AuthVersionHandler).ServeHTTP(0xc0004cd700, 0x13455c0, 0xc000518ee0, 0xc0039dad00)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/rest.go:174 +0x13d
          main.exportMuxRoutesToHttprouter.func1.1(0x13455c0, 0xc000518ee0, 0xc0039dac00, 0x0, 0x0, 0x0)
          	cbft/cmd/cbft/main.go:729 +0xa1
          github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0002a3240, 0x13455c0, 0xc000518ee0, 0xc0039dac00)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/julienschmidt/httprouter@v1.1.1-0.20170430222011-975b5c4c7c21/router.go:344 +0x932
          net/http.serverHandler.ServeHTTP(0xc000214460, 0x13455c0, 0xc000518ee0, 0xc0039dac00)
          	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
          net/http.(*conn).serve(0xc0004301e0, 0x1349ec0, 0xc001a2e200)
          	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
          created by net/http.(*Server).Serve
          	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384

          Evgeny Makarenko this test is using the upsidedown/moss index?

          Will need to be fixed regardless.

          abhinav Abhinav Dangeti added a comment - - edited Here's the full stack trace .. 2021 / 02 / 08 12 : 33 : 48 http: panic serving 127.0 . 0.1 : 36252 : runtime error: invalid memory address or nil pointer dereference goroutine 1069 [running]: net/http.(*conn).serve.func1( 0xc0004301e0 ) /home/couchbase/.cbdepscache/exploded/x86_64/go- 1.13 . 7 /go/src/net/http/server.go: 1767 + 0x139 panic( 0xffeba0 , 0x1cb22b0 ) /home/couchbase/.cbdepscache/exploded/x86_64/go- 1.13 . 7 /go/src/runtime/panic.go: 679 + 0x1b2 github.com/couchbase/moss.(*Store).Histograms(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/couchbase/moss @v0 . 1.0 /store_stats.go: 89 github.com/blevesearch/bleve/v2/index/upsidedown/store/moss.(*stats).statsMap( 0xc0000110c0 , 0x7faa69bc5a70 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/blevesearch/bleve/v2 @v2 . 0.1 /index/upsidedown/store/moss/stats.go: 48 + 0x27b github.com/blevesearch/bleve/v2/index/upsidedown/store/moss.(*Store).StatsMap( 0xc0020f2820 , 0x13431c0 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/blevesearch/bleve/v2 @v2 . 0.1 /index/upsidedown/store/moss/store.go: 214 + 0x2f github.com/blevesearch/bleve/v2/index/upsidedown.(*indexStat).statsMap( 0xc0020f2780 , 0xc0040ba780 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/blevesearch/bleve/v2 @v2 . 0.1 /index/upsidedown/stats.go: 46 + 0x478 github.com/blevesearch/bleve/v2/index/upsidedown.(*UpsideDownCouch).StatsMap( 0xc0020f0480 , 0x6 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/blevesearch/bleve/v2 @v2 . 0.1 /index/upsidedown/upsidedown.go: 1022 + 0x2f github.com/blevesearch/bleve/v2.(*IndexStat).statsMap( 0xc0020efb40 , 0xc0040ba750 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/blevesearch/bleve/v2 @v2 . 0.1 /index_stats.go: 31 + 0x4a github.com/blevesearch/bleve/v2.(*indexImpl).StatsMap( 0xc0002c8690 , 0x0 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/blevesearch/bleve/v2 @v2 . 0.1 /index_impl.go: 791 + 0x2f github.com/couchbase/cbft.(*BleveDest).StatsMap( 0xc0020ddad0 , 0xf930e0 , 0xc003eba100 , 0xc00225e270 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/pindex_bleve.go: 1911 + 0xb2 github.com/couchbase/cbft.addPIndexStats( 0xc001cd1300 , 0xc003ed9b00 , 0x11a15dd , 0x13 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/ns_server.go: 596 + 0x73 github.com/couchbase/cbft.gatherIndexStats( 0xc000422e00 , 0xc001fdf000 , 0x1 , 0xc0003d5a40 , 0x1c , 0xc0044c5460 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/ns_server.go: 350 + 0x165a github.com/couchbase/cbft.(*PrometheusHighMetricsHandler).ServeHTTP( 0xc0003e7fe0 , 0x1343c40 , 0xc003aa0760 , 0xc0039dad00 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/prometheus.go: 113 + 0x144 github.com/couchbase/cbgt/rest.(*HandlerWithRESTMeta).ServeHTTP( 0xc00050e480 , 0x13455c0 , 0xc000518ee0 , 0xc0039dad00 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/rest/rest.go: 254 + 0x10c github.com/couchbase/cbft.(*AuthVersionHandler).ServeHTTP( 0xc0004cd700 , 0x13455c0 , 0xc000518ee0 , 0xc0039dad00 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/rest.go: 174 + 0x13d main.exportMuxRoutesToHttprouter.func1. 1 ( 0x13455c0 , 0xc000518ee0 , 0xc0039dac00 , 0x0 , 0x0 , 0x0 ) cbft/cmd/cbft/main.go: 729 + 0xa1 github.com/julienschmidt/httprouter.(*Router).ServeHTTP( 0xc0002a3240 , 0x13455c0 , 0xc000518ee0 , 0xc0039dac00 ) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go- 1.13 . 7 /pkg/mod/github.com/julienschmidt/httprouter @v1 . 1.1 - 0.20170430222011 -975b5c4c7c21/router.go: 344 + 0x932 net/http.serverHandler.ServeHTTP( 0xc000214460 , 0x13455c0 , 0xc000518ee0 , 0xc0039dac00 ) /home/couchbase/.cbdepscache/exploded/x86_64/go- 1.13 . 7 /go/src/net/http/server.go: 2802 + 0xa4 net/http.(*conn).serve( 0xc0004301e0 , 0x1349ec0 , 0xc001a2e200 ) /home/couchbase/.cbdepscache/exploded/x86_64/go- 1.13 . 7 /go/src/net/http/server.go: 1890 + 0x875 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/x86_64/go- 1.13 . 7 /go/src/net/http/server.go: 2928 + 0x384 Evgeny Makarenko  this test is using the upsidedown/moss index? Will need to be fixed regardless.
          abhinav Abhinav Dangeti added a comment - - edited

          I've merged the fix for this: https://github.com/blevesearch/bleve/pull/1558

          I'll close this ticket out once i've upgraded the bleve version that cbft uses.

          abhinav Abhinav Dangeti added a comment - - edited I've merged the fix for this: https://github.com/blevesearch/bleve/pull/1558 I'll close this ticket out once i've upgraded the bleve version that cbft uses.
          abhinav Abhinav Dangeti added a comment - http://review.couchbase.org/c/cbft/+/147371

          Build couchbase-server-7.0.0-4572 contains cbft commit a52245a with commit message:
          MB-44216: Upgrade bleve version to absorb fix

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4572 contains cbft commit a52245a with commit message: MB-44216 : Upgrade bleve version to absorb fix

          Build couchbase-server-7.0.0-4603 contains cbft commit a52245a with commit message:
          MB-44216: Upgrade bleve version to absorb fix

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4603 contains cbft commit a52245a with commit message: MB-44216 : Upgrade bleve version to absorb fix

          Verified fir build 7.0.0-4617

          evgeny.makarenko Evgeny Makarenko (Inactive) added a comment - Verified fir build 7.0.0-4617

          People

            abhinav Abhinav Dangeti
            evgeny.makarenko Evgeny Makarenko (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty