Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49669

[System Test] cbq-engine panic / crash

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1742
      Test : -test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
      Scale : 3
      Iteration : 1st

      On query node 172.23.96.251, seeing a crash with the following stack trace :

      2021-11-18T17:14:24.897-08:00 [Info] switched currmeta from 3935 -> 3935 force true
      panic: runtime error: index out of range [0] with length 0
       
      goroutine 85749 [running]:
      github.com/couchbase/query/primitives/couchbase.(*Bucket).changedVBServerMap(0xc00656c5a0, 0xc007106c50, 0xb)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/primitives/couchbase/ns_server.go:352 +0x1cd
      github.com/couchbase/query/primitives/couchbase.(*Bucket).refresh(0xc00656c5a0, 0xc004eddd01, 0x25ef978, 0x12)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/primitives/couchbase/ns_server.go:1068 +0x6d1
      github.com/couchbase/query/primitives/couchbase.(*Bucket).Refresh(...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/primitives/couchbase/ns_server.go:996
      github.com/couchbase/query/primitives/couchbase.(*Pool).GetBucket(0xc004fe4a80, 0x25ef978, 0x12, 0x0, 0xc0049a7418, 0xd7cc33)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/primitives/couchbase/ns_server.go:1183 +0x76
      github.com/couchbase/query/datastore/couchbase.newKeyspace(0xc0005f1810, 0x25ef978, 0x12, 0x12, 0xc0050f8cb8, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1297 +0x6d
      github.com/couchbase/query/datastore/couchbase.(*namespace).keyspaceByName(0xc0005f1810, 0x25ef978, 0x12, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1023 +0x373
      github.com/couchbase/query/datastore/couchbase.(*store).CreateSystemCBOStats(0xc0004601c0, 0xc0069f0b70, 0x24, 0x2137060, 0x2474fa0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/system_keyspace_ee.go:55 +0xf25
      github.com/couchbase/query-ee/dictionary.CheckSysCBOStats(0x7f513959c201, 0xc0069f0b70, 0x24, 0x2474f00, 0x25d0300, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query-ee/dictionary/system_keyspace.go:102 +0x196
      github.com/couchbase/query-ee/dictionary.HasSysCBOStats(0xc004424001, 0xc0069f0b70, 0x24, 0x7f513959c2d8, 0xc004424000, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query-ee/dictionary/system_keyspace.go:62 +0x79
      github.com/couchbase/query-ee/updstat.(*DefaultStatUpdater).UpdateStatistics(0xc000414400, 0x286ecb0, 0xc00183b040, 0xc0050b9f80, 0x3, 0x4, 0x0, 0x0, 0x0, 0x28748b0, ...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query-ee/updstat/upd_statistics.go:62 +0xbd
      created by github.com/couchbase/query/execution.(*UpdateStatistics).RunOnce.func1
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/update_stat.go:87 +0x365
      2021-11-18T17:14:25.215-08:00 [INFO] Current nofiles rlimit: 200000 (max: 200000)
      2021/11/18 17:14:25 cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/manager_cluster_options_key, key: manager_cluster_options_key, deletion: false
      2021/11/18 17:14:25 cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/nodeDefs-known/36bd0f1be16a3094d2f5d4b450424594, key: nodeDefs-known/36bd0f1be16a3094d2f5d4b450424594, deletion: false
      2021-11-18T17:14:25.217-08:00 [INFO] Initialization of cbauth succeeded
      2021/11/18 17:14:25 cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/nodeDefs-known/adb09b09fc3d05fe254c1ad51ae1c1d0, key: nodeDefs-known/adb09b09fc3d05fe254c1ad51ae1c1d0, deletion: false
      2021/11/18 17:14:25 cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/nodeDefs-wanted/36bd0f1be16a3094d2f5d4b450424594, key: nodeDefs-wanted/36bd0f1be16a3094d2f5d4b450424594, deletion: false
      2021/11/18 17:14:25 cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/nodeDefs-wanted/adb09b09fc3d05fe254c1ad51ae1c1d0, key: nodeDefs-wanted/adb09b09fc3d05fe254c1ad51ae1c1d0, deletion: false
      2021/11/18 17:14:25 cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/version, key: version, deletion: false
      2021-11-18T17:14:25.721-08:00 [INFO] New store created with url http://127.0.0.1:8091
      

      This could be related to the recent changes in MB-49553. The test was updating statistics on various collections of bucket4 at this time :

      [2021-11-18T17:14:11-08:00, sequoiatools/indexmanager:53dde0] -n 172.23.108.103 -o 8091 -u Administrator -p password -b bucket4 -a enable_cbo --cbo_enable_ratio=50

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty