Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28403

[FTS-Scorch] Panic "cap out of range" from scorch.introduceMerge (scorch/introducer.go:209)

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Unknown

    Description

      Build
      5.5.0-1989

      Testcase
      ./testrunner -i /tmp/testexec.30827.ini -p cluster=D+F,index_type=scorch -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=10000,update=True,expires=30,GROUP=PS

      QE bookkeeping - test_5 in fts-sanity job

      Steps
      1. Load 10k docs in default bucket
      2. create default index, wait till indexing is done
      3. Update 30% of docs with ttl=30s.
      4. Run expiry pager after 30s. Verified that index count = bucket count = 7000.

      However, panic was found in the logs.

      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - panic: runtime error: makeslice: cap out of range
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - goroutine 17525 [running]:
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - panic(0xca6c20, 0xe581d0)
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/panic.go:540 +0x45e fp=0xc4208baba0 sp=0xc4208baaf8 pc=0x42d30e
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - runtime.makeslice(0xd43180, 0x0, 0xfffffffffffffffc, 0x0, 0xc4203c0240, 0xc4208baec0)
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/slice.go:51 +0xd2 fp=0xc4208babd0 sp=0xc4208baba0 pc=0x4454b2
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - github.com/blevesearch/bleve/index/scorch.(*Scorch).introduceMerge(0xc420374000, 0xc4243238f0)
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/introducer.go:209 +0xbb fp=0xc4208bad98 sp=0xc4208babd0 pc=0x97fcab
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - github.com/blevesearch/bleve/index/scorch.(*Scorch).mainLoop(0xc420374000)
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/introducer.go:59 +0x5b2 fp=0xc4208bafd8 sp=0xc4208bad98 pc=0x97ea92
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - runtime.goexit()
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc4208bafe0 sp=0xc4208bafd8 pc=0x45e891
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - created by github.com/blevesearch/bleve/index/scorch.(*Scorch).Open
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/scorch.go:123 +0x89
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - goroutine 1 [chan receive, 6 minutes]:
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - runtime.gopark(0xde9dc8, 0xc42032e1d8, 0xdb4c17, 0xc, 0xc420166017, 0x3)
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/proc.go:287 +0x12c fp=0xc420337b68 sp=0xc420337b38 pc=0x42f40c
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - runtime.goparkunlock(0xc42032e1d8, 0xdb4c17, 0xc, 0x1000f0100de9517, 0x3)
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/proc.go:293 +0x5e fp=0xc420337ba8 sp=0xc420337b68 pc=0x42f4fe
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - runtime.chanrecv(0xc42032e180, 0x0, 0xc89a01, 0xc4202d9d50)
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/chan.go:506 +0x304 fp=0xc420337c58 sp=0xc420337ba8 pc=0x4064d4
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - runtime.chanrecv1(0xc42032e180, 0x0)
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/chan.go:388 +0x2b fp=0xc420337c88 sp=0xc420337c58 pc=0x40617b
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - main.main()
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/cmd/cbft/main.go:226 +0x110d fp=0xc420337f80 sp=0xc420337c88 pc=0xbb91ed
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - runtime.main()
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/proc.go:195 +0x226 fp=0xc420337fe0 sp=0xc420337f80 pc=0x42ef56
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - runtime.goexit()
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420337fe8 sp=0xc420337fe0 pc=0x45e891
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - goroutine 2 [force gc (idle), 6 minutes]:
      		 
      [2018-02-26 20:52:52,666] - [fts_base:3353] INFO - runtime.gopark(0xde9dc8, 0x1414300, 0xdb70ee, 0xf, 0x14, 0x1)
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/proc.go:287 +0x12c fp=0xc42002e768 sp=0xc42002e738 pc=0x42f40c
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - runtime.goparkunlock(0x1414300, 0xdb70ee, 0xf, 0xc420000114, 0x1)
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/proc.go:293 +0x5e fp=0xc42002e7a8 sp=0xc42002e768 pc=0x42f4fe
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - runtime.forcegchelper()
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/proc.go:245 +0xcc fp=0xc42002e7e0 sp=0xc42002e7a8 pc=0x42f22c
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - runtime.goexit()
      		 
      [2018-02-26 20:52:52,667] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42002e7e8 sp=0xc42002e7e0 pc=0x45e891
      		 
      [2018-02-26 20:52:52,846] - [remote_util:232] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
      		 
      [2018-02-26 20:52:52,943] - [remote_util:266] INFO - Connected to 172.23.105.179
      		 
      [2018-02-26 20:52:53,400] - [remote_util:3046] INFO - running command.raw on 172.23.105.179: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
      		 
      [2018-02-26 20:52:53,428] - [remote_util:3083] INFO - command executed successfully
      		 
      [2018-02-26 20:52:53,428] - [fts_base:3372] INFO - 0
      		 
      [2018-02-26 20:52:53,428] - [fts_base:3380] INFO - Initial 'panic:' count on 172.23.105.179 :0, now :0
      		 
      [2018-02-26 20:52:53,542] - [fts_base:3389] ERROR - ['panic: found on 172.23.105.152']

      Testrunner log -

      [2018-02-26 20:50:52,123] - [fts_base:876] INFO - Creating fulltext-index default_index_1 on 172.23.105.152
      		 
      [2018-02-26 20:50:52,123] - [rest_client:2646] INFO - {"params": {"store": {"mossStoreOptions": {}, "kvStoreName": "mossStore", "indexType": "scorch"}}, "name": "default_index_1", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""}
      		 
      [2018-02-26 20:50:52,149] - [rest_client:2653] INFO - Index default_index_1 created
      		 
      [2018-02-26 20:50:52,149] - [fts_base:3498] INFO - Validating index distribution for default_index_1 ...
      		 
      [2018-02-26 20:50:52,180] - [fts_base:3393] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ...
      		 
      [2018-02-26 20:50:57,232] - [fts_base:3516] INFO - Validated: Number of PIndexes = 6
      		 
      [2018-02-26 20:50:57,239] - [fts_base:3528] INFO - Validated: Every pIndex serves 171 partitions or lesser
      		 
      [2018-02-26 20:50:57,239] - [fts_base:3552] INFO - Expecting num of partitions in each node in range 853-1024
      		 
      [2018-02-26 20:50:57,240] - [fts_base:3567] INFO - Validated: Node 1095b1d137973a303dc53f73345391e7 houses 6 pindexes which serve 1024 partitions
      		 
      [2018-02-26 20:50:57,299] - [rest_client:1886] INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-02-26 20:50:57,308] - [fts_base:3416] INFO - Docs in bucket = 10000, docs in FTS index 'default_index_1': 2971
      		 
      [2018-02-26 20:51:03,407] - [rest_client:1886] INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-02-26 20:51:03,415] - [fts_base:3416] INFO - Docs in bucket = 10000, docs in FTS index 'default_index_1': 6147
      		 
      [2018-02-26 20:51:09,478] - [rest_client:1886] INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-02-26 20:51:09,489] - [fts_base:3416] INFO - Docs in bucket = 10000, docs in FTS index 'default_index_1': 10000
      		 
      [2018-02-26 20:51:09,541] - [rest_client:1886] INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-02-26 20:51:09,550] - [fts_base:3690] INFO - Docs in index default_index_1=10000, bucket docs=10000
      		 
      [2018-02-26 20:51:09,550] - [fts_base:3276] INFO - Updating keys @ C1 with expiry=30
      		 
      [2018-02-26 20:51:10,195] - [data_helper:295] INFO - creating direct client 172.23.105.152:11210 default
      		 
      [2018-02-26 20:51:11,295] - [fts_base:3313] INFO - Batched updates loaded to cluster(s)
      		 
      [2018-02-26 20:51:11,296] - [fts_base:3393] INFO - sleep for 30 secs. Waiting for expiration of updated items ...
      		 
      [2018-02-26 20:51:41,350] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': '172.23.105.152', 'password': 'password', 'port': u'8091'}
      		 
      [2018-02-26 20:51:41,350] - [data_helper:293] INFO - creating direct client 172.23.105.152:11210 default
      		 
      [2018-02-26 20:51:41,395] - [cluster_helper:340] INFO - Setting flush param on server {'username': 'Administrator', 'ip': '172.23.105.152', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
      		 
      setting param: exp_pager_stime 10
      		 
      [2018-02-26 20:51:41,396] - [cluster_helper:354] INFO - Setting flush param on server {'username': 'Administrator', 'ip': '172.23.105.152', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (815168167, 0, '')
      		 
      [2018-02-26 20:51:41,396] - [fts_base:2451] INFO - wait for expiry pager to run on all these nodes
      		 
      [2018-02-26 20:51:51,406] - [fts_base:3393] INFO - sleep for 60 secs. Waiting for updates to get indexed... ...
      		 
      [2018-02-26 20:52:51,437] - [rest_client:1886] INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-02-26 20:52:51,457] - [fts_base:3416] INFO - Docs in bucket = 7000, docs in FTS index 'default_index_1': 7000
      		 
      [2018-02-26 20:52:51,474] - [rest_client:1886] INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      [2018-02-26 20:52:51,493] - [fts_base:3690] INFO - Docs in index default_index_1=7000, bucket docs=7000
      		 
      [2018-02-26 20:52:51,502] - [rest_client:1441] INFO - /diag/eval status on 172.23.105.152:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
      		 
      [2018-02-26 20:52:51,504] - [remote_util:232] INFO - connecting to 172.23.105.152 with username:root password:couchbase ssh_key:
      		 
      [2018-02-26 20:52:51,603] - [remote_util:266] INFO - Connected to 172.23.105.152
      		 
      [2018-02-26 20:52:51,990] - [remote_util:3046] INFO - running command.raw on 172.23.105.152: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
      		 
      [2018-02-26 20:52:52,011] - [remote_util:3083] INFO - command executed successfully
      		 
      [2018-02-26 20:52:52,012] - [fts_base:3372] INFO - 1
      		 
      [2018-02-26 20:52:52,012] - [fts_base:3380] INFO - Initial 'panic:' count on 172.23.105.152 :0, now :1
      		 
      [2018-02-26 20:52:52,013] - [remote_util:232] INFO - connecting to 172.23.105.152 with username:root password:couchbase ssh_key:
      		 
      [2018-02-26 20:52:52,110] - [remote_util:266] INFO - Connected to 172.23.105.152
      		 
      [2018-02-26 20:52:52,577] - [rest_client:1441] INFO - /diag/eval status on 172.23.105.152:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
      		 
      [2018-02-26 20:52:52,577] - [remote_util:3046] INFO - running command.raw on 172.23.105.152: zgrep -A 40 -B 4 'panic:' "/opt/couchbase/var/lib/couchbase/logs"/fts.log*
      		 
      [2018-02-26 20:52:52,663] - [remote_util:3083] INFO - command executed successfully
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 2018-02-26T20:51:52.430-08:00 [INFO] cbdatasource: server: 127.0.0.1:11210, uprOpenName: fts:default_index_1_3112ecfe842b52f4_54820232-6203be4e, worker, looping beg, vbucketState: "running" (has 171 vbuckets), 342-512
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 2018-02-26T20:51:52.473-08:00 [INFO] cbdatasource: server: 127.0.0.1:11210, uprOpenName: fts:default_index_1_3112ecfe842b52f4_18572d87-54f84545, worker, looping beg, vbucketState: "running" (has 171 vbuckets), 684-854
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 2018-02-26T20:51:52.510-08:00 [INFO] cbdatasource: server: 127.0.0.1:11210, uprOpenName: fts:default_index_1_3112ecfe842b52f4_aa574717-43b21d6d, worker, looping beg, vbucketState: "running" (has 171 vbuckets), 513-683
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 2018-02-26T20:51:52.531-08:00 [INFO] cbdatasource: server: 127.0.0.1:11210, uprOpenName: fts:default_index_1_3112ecfe842b52f4_6ddbfb54-3587a906, worker, looping beg, vbucketState: "running" (has 169 vbuckets), 855-1023
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - panic: runtime error: makeslice: cap out of range
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - goroutine 17525 [running]:
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - panic(0xca6c20, 0xe581d0)
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/panic.go:540 +0x45e fp=0xc4208baba0 sp=0xc4208baaf8 pc=0x42d30e
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - runtime.makeslice(0xd43180, 0x0, 0xfffffffffffffffc, 0x0, 0xc4203c0240, 0xc4208baec0)
      		 
      [2018-02-26 20:52:52,664] - [fts_base:3353] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.3/go/src/runtime/slice.go:51 +0xd2 fp=0xc4208babd0 sp=0xc4208baba0 pc=0x4454b2
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - github.com/blevesearch/bleve/index/scorch.(*Scorch).introduceMerge(0xc420374000, 0xc4243238f0)
      		 
      [2018-02-26 20:52:52,665] - [fts_base:3353] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/scorch/introducer.go:209 +0xbb fp=0xc4208bad98 sp=0xc4208babd0 pc=0x97fcab
      		 
      :
      		 
      :

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            steve Steve Yen
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty