Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49875

[System Test] cbq-engine crashes with error "panic: runtime error: index out of range [-1]"

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1822
      Test : -test tests/integration/neo/test_neo_magma_milestone3.yml -scope tests/integration/neo/scope_neo_magma.yml
      Scale : 3
      Iteration : 1st

      Seeing cbq-engine crash on multiple nodes multiple times with the following stack trace :

      This is from the node 172.23.96.48, but the crashes are there on other query nodes as well - 172.23.96.254, 172.23.97.112.

      2021-12-02T01:00:40.190-08:00 [Info] GSIC[default/bucket5-scope_0-coll_3-1638435400165951464] logstats "bucket5" {"gsi_scan_count":7,"gsi_scan_duration":26670421956,"gsi_throttle_duration":1394237936,"gsi_prime_duration":170473496,"gsi_blocked_duration":21840893003,"gsi_total_temp_files":16}
      panic: runtime error: index out of range [-1]
       
      goroutine 620415 [running]:
      github.com/couchbase/query/inferencer.(*UnifiedDocumentRetriever).GetNextDoc(0xc016f42400, 0x28702c0, 0xc000160780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/inferencer/document_retrievers.go:657 +0x1b2a
      github.com/couchbase/query-ee/updstat.doUpdateStatistics(0x28782d0, 0xc0001efe10, 0x2847f80, 0xc016f42400, 0xc01b6374e0, 0x5, 0x8, 0x0, 0x6b0a3, 0xc002fc3a70, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query-ee/updstat/upd_statistics.go:370 +0x1ad
      github.com/couchbase/query-ee/updstat.(*DefaultStatUpdater).UpdateStatistics(0xc00037e8d0, 0x28782d0, 0xc0001efe10, 0xc000fa1e80, 0x3, 0x4, 0xc01b637440, 0xf, 0x12, 0x287def0, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query-ee/updstat/upd_statistics.go:290 +0x9ac
      created by github.com/couchbase/query/execution.(*UpdateStatistics).RunOnce.func1
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/update_stat.go:87 +0x365
      2021-12-02T01:00:42.002-08:00 [INFO] Current nofiles rlimit: 200000 (max: 200000)
      

      Last known good build : 7.1.0-1809
      Possible change that might have caused this regression:

      Commit: eedbcdd25937df6fa0eb730bf9b30d6837b591dd in build: couchbase-server-7.1.0-1822
      MB-49860 Need to reset document retriever if multiple passes needed for UPDATE STATISTICS

      Change-Id: I56c09a5f96f7ac923020590f1394417486d01c21
      Reviewed-on: https://review.couchbase.org/c/query-ee/+/166792
      Reviewed-by: Donald Haggart <donald.haggart@couchbase.com>
      Tested-by: Bingjie Miao <bingjie.miao@couchbase.com>
      Author: miaobingjie <bingjie.miao@couchbase.com>
      Committer: Bingjie Miao <bingjie.miao@couchbase.com>

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - cc Donald Haggart

          Build couchbase-server-7.1.0-1835 contains query commit eddde07 with commit message:
          MB-49875 Close index connection on retriever reset and directly indicate when cache is active and should be used.

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1835 contains query commit eddde07 with commit message: MB-49875 Close index connection on retriever reset and directly indicate when cache is active and should be used.

          Mihir Kamdar , I'm marking as resolved as I believe it is, but of course the acid test will be a re-run...

          Donald.haggart Donald Haggart added a comment - Mihir Kamdar , I'm marking as resolved as I believe it is, but of course the acid test will be a re-run...

          Build couchbase-server-7.1.0-1837 contains query commit c981ab0 with commit message:
          MB-49875 Revise fix.

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1837 contains query commit c981ab0 with commit message: MB-49875 Revise fix.

          Closing as this issue wasnt seen after the fix. Last checked on 7.1.0-1997.

          mihir.kamdar Mihir Kamdar added a comment - Closing as this issue wasnt seen after the fix. Last checked on 7.1.0-1997.

          People

            mihir.kamdar Mihir Kamdar
            mihir.kamdar Mihir Kamdar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty