Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37194

Panic in query logs while running eventing performance test

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      Build 6.5.0-4926

      Observed panic "invalid memory address or nil pointer dereference" in query.log file while running eventing performance test with n1ql query.
      Eventing code-

      function OnUpdate(doc, meta) {
      	var docId = meta.id;
      	var select_query = SELECT * FROM `bucket-1` USE KEYS[$docId];
      	for (var r of select_query) {
      	}
      	select_query.close()
      	var upsert_query = UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, 'Hello World');
      	upsert_query.close();
      }
      

      Cluster configuration-

      172.23.96.16:kv,n1ql
      172.23.96.17:kv,n1ql
      172.23.96.20:kv,n1ql
      172.23.96.23:kv,n1ql
      172.23.97.177:eventing

      Note that this is not consistently reproducible and rerun of same test does not show this panic, also last weekly build 6.5.0-4908 does not show this panic.

      Job where we see panic- http://perf.jenkins.couchbase.com/job/themis/4557
      Logs-
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.16.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.17.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.20.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.23.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.97.177.zip
      Node 172.23.96.17 has this panic.

      Panic-

      2019/12/06 12:27:21 http: panic serving 172.23.96.16:59602: runtime error: invalid memory address or nil pointer dereference
      goroutine 119 [running]:
      net/http.(*conn).serve.func1(0xc001820280)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0
      panic(0x1ca8800, 0x3110490)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9
      github.com/couchbase/query/datastore/couchbase.(*keyspace).Namespace(0x0, 0x1, 0x2)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1172 +0x5
      github.com/couchbase/query/plan.verifyKeyspace(0x20e3b40, 0x0, 0xc001c02000, 0x0, 0x7f3a71fa96c0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/verify.go:45 +0x35
      github.com/couchbase/query/plan.(*SendUpsert).verify(0xc0016b9c00, 0xc001c02000, 0xc001d477b0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/upsert.go:122 +0x41
      github.com/couchbase/query/plan.(*Sequence).verify(0xc00165ee40, 0xc001c02000, 0xc001d477f8)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/sequence.go:94 +0x6f
      github.com/couchbase/query/plan.(*Parallel).verify(0xc000609500, 0xc001c02000, 0x1)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/parallel.go:95 +0x3d
      github.com/couchbase/query/plan.(*Sequence).verify(0xc0013d27c0, 0xc001c02000, 0x24)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/sequence.go:94 +0x6f
      github.com/couchbase/query/plan.(*Authorize).verify(0xc001548580, 0xc001c02000, 0xd2a89e)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/authorize.go:90 +0x3e
      github.com/couchbase/query/plan.(*Sequence).verify(0xc001394340, 0xc001c02000, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/sequence.go:94 +0x6f
      github.com/couchbase/query/plan.(*Prepared).Verify(0xc001c02000, 0xc00164a210)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/prepared.go:247 +0x38
      github.com/couchbase/query/prepareds.DecodePrepared(0xc0016d0515, 0x24, 0xc001a0e900, 0x224, 0xc001a00000, 0x0, 0xc000812960, 0x2d3433392e30323a, 0x30303a3830)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:580 +0x2ae
      github.com/couchbase/query/server/http.doPrepared(0xc000346000, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00, 0xc0016b6d00, 0x0, 0x0, 0x0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_accounting_endpoint.go:304 +0x4b3
      github.com/couchbase/query/server/http.(*HttpEndpoint).wrapAPI(0xc000346000, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00, 0x1f69450)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_endpoint.go:37 +0x187
      github.com/couchbase/query/server/http.(*HttpEndpoint).registerAccountingHandlers.func5(0x20cd520, 0xc0015ce1c0, 0xc0016b6c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_accounting_endpoint.go:64 +0x54
      net/http.HandlerFunc.ServeHTTP(0xc00002abb0, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1964 +0x44
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc000fc80f0, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/gorilla/mux/mux.go:114 +0xe0
      net/http.serverHandler.ServeHTTP(0xc0008288f0, 0x20cd520, 0xc0015ce1c0, 0xc0016b6a00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2741 +0xab
      net/http.(*conn).serve(0xc001820280, 0x20d3260, 0xc000436bc0)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1847 +0x646
      created by net/http.(*Server).Serve
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2851 +0x2f5
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

          No query changes went in after 6.5.0-4908

          Marco Greco Not sure how pointer reset in prepared statement?

          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited No query changes went in after 6.5.0-4908 Marco Greco Not sure how pointer reset in prepared statement?
          marco.greco Marco Greco added a comment - - edited

          The stack provided was incomplete and in fact misleading.
          Complete stack now provided, and the issue is a prepared UPSERT plan being sent to other nodes.
          The plan contains a keyspace that does not exist - issue is that upsert.UnmarshalJSON() does not return an error if the keyspace is not found, and instead it sets a nil keyspace.
          We then panic trying to verify it.
          Issue has existed since upsert was coded and exposed in 5.5 with auto_reprepare.

          marco.greco Marco Greco added a comment - - edited The stack provided was incomplete and in fact misleading. Complete stack now provided, and the issue is a prepared UPSERT plan being sent to other nodes. The plan contains a keyspace that does not exist - issue is that upsert.UnmarshalJSON() does not return an error if the keyspace is not found, and instead it sets a nil keyspace. We then panic trying to verify it. Issue has existed since upsert was coded and exposed in 5.5 with auto_reprepare.

          Build couchbase-server-6.5.0-4938 contains query commit a1708ed with commit message:
          MB-37194 panic when distributing upsert plan

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4938 contains query commit a1708ed with commit message: MB-37194 panic when distributing upsert plan

          Build couchbase-server-7.0.0-1120 contains query commit ae431e5 with commit message:
          MB-37194 panic when distributing upsert plan

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1120 contains query commit ae431e5 with commit message: MB-37194 panic when distributing upsert plan

          Build couchbase-server-6.5.1-6005 contains query commit a1708ed with commit message:
          MB-37194 panic when distributing upsert plan

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains query commit a1708ed with commit message: MB-37194 panic when distributing upsert plan

          Marco Greco,
          Is there anyway to reproduce this issue consistently to verify. Thanks.

          mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Marco Greco , Is there anyway to reproduce this issue consistently to verify. Thanks.
          marco.greco Marco Greco added a comment -

          it's not an easy reproduction, but

          have a cluster with 4 n1ql nodes
          in a loop do prepare upsert blah blah blah like in the event function, drop the bucket, delete from systm:prepareds, create the bucket
          you should have errors in the query logs, along the lines of could no distribute prepare statement....
          but never panics

          marco.greco Marco Greco added a comment - it's not an easy reproduction, but have a cluster with 4 n1ql nodes in a loop do prepare upsert blah blah blah like in the event function, drop the bucket, delete from systm:prepareds, create the bucket you should have errors in the query logs, along the lines of could no distribute prepare statement.... but never panics

          Build 6.5.0-4938

          Reran the same eventing test 3 times and not observed this issue.

          Jobs-

          http://perf.jenkins.couchbase.com/job/themis/4588/

          http://perf.jenkins.couchbase.com/job/themis/4589/

          http://perf.jenkins.couchbase.com/job/themis/4590/

           

          Also tried to delete the bucket and delete from system:prepareds and no panics observed.

           

          mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Build 6.5.0-4938 Reran the same eventing test 3 times and not observed this issue. Jobs- http://perf.jenkins.couchbase.com/job/themis/4588/ http://perf.jenkins.couchbase.com/job/themis/4589/ http://perf.jenkins.couchbase.com/job/themis/4590/   Also tried to delete the bucket and delete from system:prepareds and no panics observed.  

          People

            mahesh.mandhare Mahesh Mandhare (Inactive)
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty