Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44240

[System Test] : Seeing several panics in query logs while inserting docs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • query

    Description

      Build : 7.0.0-4432
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st

      Just after starting the tpcc workload in the longevity system test, across both the query nodes, we are observing several panics.

      2021-02-10T10:04:00.206-08:00 [Warn] Scan failed with error for index 16393101330221762192.  Trying scan again with replica, reqId:eb93ffb0-3b5a-4c6a-8618-05a47df543ee :  Indexer rollback or warmup from [172.23.104.155:9101 172.23.97.242:9101 172.23.123.28:9101] ...
      		 
      2021-02-10T10:04:00.222-08:00 [Info] GSIC[default/default-_default-_default-1612960937570106591] bbb8ea3c-2845-44cc-9eb1-816855800825 new temp file ... /opt/couchbase/var/lib/couchbase/tmp/scan-results29158525881233
      		 
      2021-02-10T10:04:00.236-08:00 [Info] GSIC[default/default-_default-_default-1612960937570106591] 80bf8abe-72a1-479c-9266-69bfb962e7f7 new temp file ... /opt/couchbase/var/lib/couchbase/tmp/scan-results29158742568892
      		 
      _time=2021-02-10T10:04:00.262-08:00 _level=SEVERE _msg=panic: runtime error: invalid memory address or nil pointer dereference
      		 
      _time=2021-02-10T10:04:00.262-08:00 _level=SEVERE _msg=request text: <ud>INSERT INTO HISTORY(KEY, VALUE) VALUES (TO_STRING($1)|| '.' || TO_STRING($2)|| '.' || TO_STRING($3), {"H_C_ID":$1, "H_C_D_ID":$2, "H_C_W_ID":$3, "H_D_ID":$4, "H_W_ID":$5, "H_DATE":$6, "H_AMOUNT":$7, "H_DATA":$8})</ud>
      		 
      _time=2021-02-10T10:04:00.262-08:00 _level=SEVERE _msg=query context: <ud></ud>
      		 
      _time=2021-02-10T10:04:00.262-08:00 _level=SEVERE _msg=stack: goroutine 22928484 [running]:
      		 
      github.com/couchbase/query/execution.(*Context).Recover(0xc1f1df8280, 0xc074d73200)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/context.go:913 +0xbb
      		 
      panic(0x222c580, 0x39046d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/query/execution.(*base).close(0xc074d73200, 0xc1f1df8280)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:265 +0x22a
      		 
      panic(0x222c580, 0x39046d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/query/execution.(*base).notify(0xc074d73200)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:949 +0x6e
      		 
      panic(0x222c580, 0x39046d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/go-couchbase.(*Bucket).WriteWithCAS(0xc0045b30e0, 0xc1f21b98f0, 0x7, 0x0, 0x0, 0x21f8440, 0xc1f21b7620, 0x2, 0xc17c6a3a38, 0x1, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/go-couchbase/client.go:987 +0x1e3
      		 
      github.com/couchbase/go-couchbase.(*Bucket).AddWithCAS(0xc0045b30e0, 0xc1f21b98f0, 0x7, 0x0, 0x21f8440, 0xc1f21b7620, 0xc17c6a3a38, 0x1, 0x1, 0x400, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/go-couchbase/client.go:1175 +0xba
      		 
      github.com/couchbase/query/datastore/couchbase.(*keyspace).performOp(0xc000b18000, 0x1, 0xc1eff65f20, 0x21, 0xc002dbb6e8, 0x8, 0xc002dbb6e0, 0x8, 0xc040ef5400, 0x1, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1861 +0x119b
      		 
      github.com/couchbase/query/datastore/couchbase.(*collection).Insert(0xc003388750, 0xc040ef5400, 0x1, 0x40, 0x27fcc00, 0xc1f1df8280, 0xc1f1df8280, 0x285c6e0, 0xc1f218c900, 0x0, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/collections.go:323 +0x1ba
      		 
      github.com/couchbase/query/execution.(*SendInsert).flushBatch(0xc074d73200, 0xc1f1df8280, 0xc1f2171800)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/insert_send.go:203 +0x5bc
      		 
      github.com/couchbase/query/execution.(*SendInsert).afterItems(0xc074d73200, 0xc1f1df8280)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/insert_send.go:106 +0x35
      		 
      github.com/couchbase/query/execution.(*base).runConsumer.func1()
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:880 +0x359
      		 
      github.com/couchbase/query/util.(*Once).Do(0xc074d73338, 0xc17c6a3f00)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:57 +0x4a
      		 
      github.com/couchbase/query/execution.(*base).runConsumer(0xc074d73200, 0x27ebda0, 0xc074d73200, 0xc1f1df8280, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:821 +0xa2
      		 
      github.com/couchbase/query/execution.(*SendInsert).RunOnce(0xc074d73200, 0xc1f1df8280, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/insert_send.go:65 +0x5c
      		 
      github.com/couchbase/query/execution.execOp(0x28398a0, 0xc074d73200, 0xc1f1df8280, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:499 +0x54
      		 
      created by github.com/couchbase/query/execution.(*base).fork
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:509 +0xfa
      		 
       
      		 
      goroutine 22928484 [running]:
      		 
      github.com/couchbase/query/execution.(*Context).Recover(0xc1f1df8280, 0xc074d73200)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/context.go:913 +0xbb
      		 
      panic(0x222c580, 0x39046d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/query/execution.(*base).close(0xc074d73200, 0xc1f1df8280)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:265 +0x22a
      		 
      panic(0x222c580, 0x39046d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/query/execution.(*base).notify(0xc074d73200)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:949 +0x6e
      		 
      panic(0x222c580, 0x39046d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/go-couchbase.(*Bucket).WriteWithCAS(0xc0045b30e0, 0xc1f21b98f0, 0x7, 0x0, 0x0, 0x21f8440, 0xc1f21b7620, 0x2, 0xc17c6a3a38, 0x1, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/go-couchbase/client.go:987 +0x1e3
      		 
      github.com/couchbase/go-couchbase.(*Bucket).AddWithCAS(0xc0045b30e0, 0xc1f21b98f0, 0x7, 0x0, 0x21f8440, 0xc1f21b7620, 0xc17c6a3a38, 0x1, 0x1, 0x400, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/go-couchbase/client.go:1175 +0xba
      		 
      github.com/couchbase/query/datastore/couchbase.(*keyspace).performOp(0xc000b18000, 0x1, 0xc1eff65f20, 0x21, 0xc002dbb6e8, 0x8, 0xc002dbb6e0, 0x8, 0xc040ef5400, 0x1, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1861 +0x119b
      		 
      github.com/couchbase/query/datastore/couchbase.(*collection).Insert(0xc003388750, 0xc040ef5400, 0x1, 0x40, 0x27fcc00, 0xc1f1df8280, 0xc1f1df8280, 0x285c6e0, 0xc1f218c900, 0x0, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/collections.go:323 +0x1ba
      		 
      github.com/couchbase/query/execution.(*SendInsert).flushBatch(0xc074d73200, 0xc1f1df8280, 0xc1f2171800)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/insert_send.go:203 +0x5bc
      		 
      github.com/couchbase/query/execution.(*SendInsert).afterItems(0xc074d73200, 0xc1f1df8280)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/insert_send.go:106 +0x35
      		 
      github.com/couchbase/query/execution.(*base).runConsumer.func1()
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:880 +0x359
      		 
      github.com/couchbase/query/util.(*Once).Do(0xc074d73338, 0xc17c6a3f00)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:57 +0x4a
      		 
      github.com/couchbase/query/execution.(*base).runConsumer(0xc074d73200, 0x27ebda0, 0xc074d73200, 0xc1f1df8280, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:821 +0xa2
      		 
      github.com/couchbase/query/execution.(*SendInsert).RunOnce(0xc074d73200, 0xc1f1df8280, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/insert_send.go:65 +0x5c
      		 
      github.com/couchbase/query/execution.execOp(0x28398a0, 0xc074d73200, 0xc1f1df8280, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:499 +0x54
      		 
      created by github.com/couchbase/query/execution.(*base).fork
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:509 +0xfa

      cbq-engine doesnt crash because of these.

      There was one unhealthy KV node in the cluster at the time, 172.23.106.100, if that is causing these errors. But then again, Query should be resilient to such situation.

      Nodes with n1ql service : 172.23.96.251, 172.23.96.252
      Nodes with index service : 172.23.104.155, 172.23.104.70, 172.23.121.117, 172.23.123.28, 172.23.96.253, 172.23.97.242

      Attachments

        Issue Links

          is triggered by

          Task - A task that needs to be done. MB-43850 Populate Cas value from successful response of UPDATE/INSERT/UPSERT/MERGE

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty