Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37194

Panic in query logs while running eventing performance test

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      Build 6.5.0-4926

      Observed panic "invalid memory address or nil pointer dereference" in query.log file while running eventing performance test with n1ql query.
      Eventing code-

      function OnUpdate(doc, meta) {
      	var docId = meta.id;
      	var select_query = SELECT * FROM `bucket-1` USE KEYS[$docId];
      	for (var r of select_query) {
      	}
      	select_query.close()
      	var upsert_query = UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, 'Hello World');
      	upsert_query.close();
      }
      

      Cluster configuration-

      172.23.96.16:kv,n1ql
      172.23.96.17:kv,n1ql
      172.23.96.20:kv,n1ql
      172.23.96.23:kv,n1ql
      172.23.97.177:eventing

      Note that this is not consistently reproducible and rerun of same test does not show this panic, also last weekly build 6.5.0-4908 does not show this panic.

      Job where we see panic- http://perf.jenkins.couchbase.com/job/themis/4557
      Logs-
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.16.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.17.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.20.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.96.23.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-4557/172.23.97.177.zip
      Node 172.23.96.17 has this panic.

      Panic-

      2019/12/06 12:27:21 http: panic serving 172.23.96.16:59602: runtime error: invalid memory address or nil pointer dereference
      goroutine 119 [running]:
      net/http.(*conn).serve.func1(0xc001820280)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0
      panic(0x1ca8800, 0x3110490)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9
      github.com/couchbase/query/datastore/couchbase.(*keyspace).Namespace(0x0, 0x1, 0x2)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1172 +0x5
      github.com/couchbase/query/plan.verifyKeyspace(0x20e3b40, 0x0, 0xc001c02000, 0x0, 0x7f3a71fa96c0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/verify.go:45 +0x35
      github.com/couchbase/query/plan.(*SendUpsert).verify(0xc0016b9c00, 0xc001c02000, 0xc001d477b0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/upsert.go:122 +0x41
      github.com/couchbase/query/plan.(*Sequence).verify(0xc00165ee40, 0xc001c02000, 0xc001d477f8)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/sequence.go:94 +0x6f
      github.com/couchbase/query/plan.(*Parallel).verify(0xc000609500, 0xc001c02000, 0x1)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/parallel.go:95 +0x3d
      github.com/couchbase/query/plan.(*Sequence).verify(0xc0013d27c0, 0xc001c02000, 0x24)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/sequence.go:94 +0x6f
      github.com/couchbase/query/plan.(*Authorize).verify(0xc001548580, 0xc001c02000, 0xd2a89e)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/authorize.go:90 +0x3e
      github.com/couchbase/query/plan.(*Sequence).verify(0xc001394340, 0xc001c02000, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/sequence.go:94 +0x6f
      github.com/couchbase/query/plan.(*Prepared).Verify(0xc001c02000, 0xc00164a210)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/plan/prepared.go:247 +0x38
      github.com/couchbase/query/prepareds.DecodePrepared(0xc0016d0515, 0x24, 0xc001a0e900, 0x224, 0xc001a00000, 0x0, 0xc000812960, 0x2d3433392e30323a, 0x30303a3830)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:580 +0x2ae
      github.com/couchbase/query/server/http.doPrepared(0xc000346000, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00, 0xc0016b6d00, 0x0, 0x0, 0x0, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_accounting_endpoint.go:304 +0x4b3
      github.com/couchbase/query/server/http.(*HttpEndpoint).wrapAPI(0xc000346000, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00, 0x1f69450)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_endpoint.go:37 +0x187
      github.com/couchbase/query/server/http.(*HttpEndpoint).registerAccountingHandlers.func5(0x20cd520, 0xc0015ce1c0, 0xc0016b6c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_accounting_endpoint.go:64 +0x54
      net/http.HandlerFunc.ServeHTTP(0xc00002abb0, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1964 +0x44
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc000fc80f0, 0x20cd520, 0xc0015ce1c0, 0xc0016b6c00)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/gorilla/mux/mux.go:114 +0xe0
      net/http.serverHandler.ServeHTTP(0xc0008288f0, 0x20cd520, 0xc0015ce1c0, 0xc0016b6a00)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2741 +0xab
      net/http.(*conn).serve(0xc001820280, 0x20d3260, 0xc000436bc0)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1847 +0x646
      created by net/http.(*Server).Serve
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2851 +0x2f5
      

      Attachments

        For Gerrit Dashboard: MB-37194
        # Subject Branch Project Status CR V

        Activity

          Build 6.5.0-4938

          Reran the same eventing test 3 times and not observed this issue.

          Jobs-

          http://perf.jenkins.couchbase.com/job/themis/4588/

          http://perf.jenkins.couchbase.com/job/themis/4589/

          http://perf.jenkins.couchbase.com/job/themis/4590/

           

          Also tried to delete the bucket and delete from system:prepareds and no panics observed.

           

          mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Build 6.5.0-4938 Reran the same eventing test 3 times and not observed this issue. Jobs- http://perf.jenkins.couchbase.com/job/themis/4588/ http://perf.jenkins.couchbase.com/job/themis/4589/ http://perf.jenkins.couchbase.com/job/themis/4590/   Also tried to delete the bucket and delete from system:prepareds and no panics observed.  
          marco.greco Marco Greco added a comment -

          it's not an easy reproduction, but

          have a cluster with 4 n1ql nodes
          in a loop do prepare upsert blah blah blah like in the event function, drop the bucket, delete from systm:prepareds, create the bucket
          you should have errors in the query logs, along the lines of could no distribute prepare statement....
          but never panics

          marco.greco Marco Greco added a comment - it's not an easy reproduction, but have a cluster with 4 n1ql nodes in a loop do prepare upsert blah blah blah like in the event function, drop the bucket, delete from systm:prepareds, create the bucket you should have errors in the query logs, along the lines of could no distribute prepare statement.... but never panics

          Marco Greco,
          Is there anyway to reproduce this issue consistently to verify. Thanks.

          mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Marco Greco , Is there anyway to reproduce this issue consistently to verify. Thanks.

          Build couchbase-server-6.5.1-6005 contains query commit a1708ed with commit message:
          MB-37194 panic when distributing upsert plan

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains query commit a1708ed with commit message: MB-37194 panic when distributing upsert plan

          Build couchbase-server-7.0.0-1120 contains query commit ae431e5 with commit message:
          MB-37194 panic when distributing upsert plan

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1120 contains query commit ae431e5 with commit message: MB-37194 panic when distributing upsert plan

          Build couchbase-server-6.5.0-4938 contains query commit a1708ed with commit message:
          MB-37194 panic when distributing upsert plan

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4938 contains query commit a1708ed with commit message: MB-37194 panic when distributing upsert plan
          marco.greco Marco Greco added a comment - - edited

          The stack provided was incomplete and in fact misleading.
          Complete stack now provided, and the issue is a prepared UPSERT plan being sent to other nodes.
          The plan contains a keyspace that does not exist - issue is that upsert.UnmarshalJSON() does not return an error if the keyspace is not found, and instead it sets a nil keyspace.
          We then panic trying to verify it.
          Issue has existed since upsert was coded and exposed in 5.5 with auto_reprepare.

          marco.greco Marco Greco added a comment - - edited The stack provided was incomplete and in fact misleading. Complete stack now provided, and the issue is a prepared UPSERT plan being sent to other nodes. The plan contains a keyspace that does not exist - issue is that upsert.UnmarshalJSON() does not return an error if the keyspace is not found, and instead it sets a nil keyspace. We then panic trying to verify it. Issue has existed since upsert was coded and exposed in 5.5 with auto_reprepare.
          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

          No query changes went in after 6.5.0-4908

          Marco Greco Not sure how pointer reset in prepared statement?

          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited No query changes went in after 6.5.0-4908 Marco Greco Not sure how pointer reset in prepared statement?

          People

            mahesh.mandhare Mahesh Mandhare (Inactive)
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty