Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36363

[high-bucket] Panic in query logs for hight bucket density test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.5.0
    • 5.5.5, 6.0.3, 6.5.0
    • query
    • 6.5.0-4380
    • Untriaged
    • Unknown

    Description

      Build 6.5.0-4380

      Observed following panic in query logs while running high bucket density test.

      Note that this panic seen only once. Here are steps I did when we see panic-

      • In this case, index+n1ql swap rebalance failed in test.
      • I tried manually taking 172.23.97.15 node out and rebalance.
      • I stopped rebalance before it completes.
      • I made 172.23.97.15 node hard fail over and issued rebalance, this complex successfully. But we see panics in query logs.

      Panic-

      2019/10/07 02:10:10 http: panic serving 172.23.97.16:55062: runtime error: invalid memory address or nil pointer dereference
      goroutine 4540 [running]:
      net/http.(*conn).serve.func1(0xc002d86500)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0
      panic(0x1cfc080, 0x320d490)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9
      github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc00012e640, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:304 +0x8db
      github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc00012e640, 0x0, 0x0, 0x0, 0xc003f52a50)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:263 +0x2f
      github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc000674b40, 0x2147560, 0xc003f52a60)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:566 +0x1e2
      github.com/couchbase/query/prepareds.(*preparedCache).getPrepared(0x3427f90, 0x2147560, 0xc003f52a10, 0x7, 0xc003f4eeb8, 0x37, 0x1c70ba0, 0xc003f52a20)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:443 +0x5a1
      github.com/couchbase/query/prepareds.GetPrepared(0x2147560, 0xc003f52a10, 0xc000000007, 0xc003f4eeb8, 0xc003f529c0, 0x2147560, 0xc003f52a10)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:424 +0x53
      github.com/couchbase/query/server/http.getPrepared(0x213c100, 0xc003f50f80, 0xc003f4eeb0, 0x8, 0x1c70ba0, 0xc003f529c0, 0xc003f4eeb8, 0xc003ec79d8, 0xca804e, 0xc72591, ...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:620 +0x12f
      github.com/couchbase/query/server/http.handlePrepared(0xc0034ad800, 0x213c100, 0xc003f50f80, 0xc003f4eeb0, 0x8, 0x1c70ba0, 0xc003f529c0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:191 +0xa3
      github.com/couchbase/query/server/http.(*jsonArgs).processParameters(0xc003f50f80, 0xc0034ad800, 0xd, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:1170 +0x158
      github.com/couchbase/query/server/http.newHttpRequest(0xc0034ad800, 0x2129760, 0xc002923500, 0xc00316d200, 0x21281e0, 0xc001e68000, 0x4000000, 0xc000406360)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:116 +0x778
      github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc001e9a000, 0x2129760, 0xc002923500, 0xc00316d200)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:153 +0x80
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc001e66050, 0x2129760, 0xc002923500, 0xc00316d200)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/gorilla/mux/mux.go:114 +0xe0
      net/http.serverHandler.ServeHTTP(0xc001ba9ee0, 0x2129760, 0xc002923500, 0xc00316d000)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2741 +0xab
      net/http.(*conn).serve(0xc002d86500, 0x212f1e0, 0xc003607e80)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1847 +0x646
      created by net/http.(*Server).Serve
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2851 +0x2f5
      ...
      

      Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/320
      Logs-
      Index+n1ql nodes-
      Panic in logs- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.15.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.19.zip
      Panic in logs-https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.20.zip
      Other nodes-
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.16.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.17.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.20.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.23.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.12.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.13.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.14.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.177.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mahesh.mandhare Mahesh Mandhare (Inactive)
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty