Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36363

[high-bucket] Panic in query logs for hight bucket density test

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.5.0, 6.0.3, 5.5.5
    • 6.5.0
    • query
    • 6.5.0-4380
    • Untriaged
    • Unknown

    Description

      Build 6.5.0-4380

      Observed following panic in query logs while running high bucket density test.

      Note that this panic seen only once. Here are steps I did when we see panic-

      • In this case, index+n1ql swap rebalance failed in test.
      • I tried manually taking 172.23.97.15 node out and rebalance.
      • I stopped rebalance before it completes.
      • I made 172.23.97.15 node hard fail over and issued rebalance, this complex successfully. But we see panics in query logs.

      Panic-

      2019/10/07 02:10:10 http: panic serving 172.23.97.16:55062: runtime error: invalid memory address or nil pointer dereference
      goroutine 4540 [running]:
      net/http.(*conn).serve.func1(0xc002d86500)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0
      panic(0x1cfc080, 0x320d490)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9
      github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc00012e640, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:304 +0x8db
      github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc00012e640, 0x0, 0x0, 0x0, 0xc003f52a50)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:263 +0x2f
      github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc000674b40, 0x2147560, 0xc003f52a60)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:566 +0x1e2
      github.com/couchbase/query/prepareds.(*preparedCache).getPrepared(0x3427f90, 0x2147560, 0xc003f52a10, 0x7, 0xc003f4eeb8, 0x37, 0x1c70ba0, 0xc003f52a20)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:443 +0x5a1
      github.com/couchbase/query/prepareds.GetPrepared(0x2147560, 0xc003f52a10, 0xc000000007, 0xc003f4eeb8, 0xc003f529c0, 0x2147560, 0xc003f52a10)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:424 +0x53
      github.com/couchbase/query/server/http.getPrepared(0x213c100, 0xc003f50f80, 0xc003f4eeb0, 0x8, 0x1c70ba0, 0xc003f529c0, 0xc003f4eeb8, 0xc003ec79d8, 0xca804e, 0xc72591, ...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:620 +0x12f
      github.com/couchbase/query/server/http.handlePrepared(0xc0034ad800, 0x213c100, 0xc003f50f80, 0xc003f4eeb0, 0x8, 0x1c70ba0, 0xc003f529c0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:191 +0xa3
      github.com/couchbase/query/server/http.(*jsonArgs).processParameters(0xc003f50f80, 0xc0034ad800, 0xd, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:1170 +0x158
      github.com/couchbase/query/server/http.newHttpRequest(0xc0034ad800, 0x2129760, 0xc002923500, 0xc00316d200, 0x21281e0, 0xc001e68000, 0x4000000, 0xc000406360)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:116 +0x778
      github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc001e9a000, 0x2129760, 0xc002923500, 0xc00316d200)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:153 +0x80
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc001e66050, 0x2129760, 0xc002923500, 0xc00316d200)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/gorilla/mux/mux.go:114 +0xe0
      net/http.serverHandler.ServeHTTP(0xc001ba9ee0, 0x2129760, 0xc002923500, 0xc00316d000)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2741 +0xab
      net/http.(*conn).serve(0xc002d86500, 0x212f1e0, 0xc003607e80)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1847 +0x646
      created by net/http.(*Server).Serve
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:2851 +0x2f5
      ...
      

      Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/320
      Logs-
      Index+n1ql nodes-
      Panic in logs- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.15.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.19.zip
      Panic in logs-https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.20.zip
      Other nodes-
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.16.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.17.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.20.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.96.23.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.12.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.13.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.14.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-320/172.23.97.177.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          mahesh.mandhare Mahesh Mandhare (Inactive) created issue -

          Build 6.5.0-4380

          Observed similar panic when index+n1ql node swap rebalance completed successfully as well.
          Below is panic and logs.

          Panic-

          2019/10/08 05:20:35 http: panic serving 172.23.97.16:40064: runtime error: invalid memory address or nil pointer dereference
          goroutine 4088 [running]:
          net/http.(*conn).serve.func1(0xc00245aaa0)
           /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0
          panic(0x1cfc080, 0x320d490)
           /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9
          github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc0014500a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:304 +0x8db
          github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc0014500a0, 0x0, 0x0, 0x0, 0xc000809fa0)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:263 +0x2f
          github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc0011c5e60, 0x2147560, 0xc000809fb0)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:566 +0x1e2
          github.com/couchbase/query/prepareds.(*preparedCache).getPrepared(0x3427f90, 0x2147560, 0xc000809f60, 0x7, 0xc002753068, 0x37, 0x1c70ba0, 0xc000809f70)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:443 +0x5a1
          github.com/couchbase/query/prepareds.GetPrepared(0x2147560, 0xc000809f60, 0xc000000007, 0xc002753068, 0xc000809ef0, 0x2147560, 0xc000809f60)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:424 +0x53
          github.com/couchbase/query/server/http.getPrepared(0x213c100, 0xc00231eae0, 0xc002753060, 0x8, 0x1c70ba0, 0xc000809ef0, 0xc002753068, 0xc0024cd9d8, 0x100000000ca804e, 0xc72591, ...)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:620 +0x12f
          github.com/couchbase/query/server/http.handlePrepared(0xc001ca2000, 0x213c100, 0xc00231eae0, 0xc002753060, 0x8, 0x1c70ba0, 0xc000809ef0, 0x0, 0x0)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:191 +0xa3
          github.com/couchbase/query/server/http.(*jsonArgs).processParameters(0xc00231eae0, 0xc001ca2000, 0xd, 0x0)
           /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:1170 +0x158
          ...
          

           
          Logs-
          Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/322/
          Logs-
          Panic in logs- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.15.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.20.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.16.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.17.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.23.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.12.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.13.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.14.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.177.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.19.zip
          https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.20.zip
           

          mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Build 6.5.0-4380 Observed similar panic when index+n1ql node swap rebalance completed successfully as well. Below is panic and logs. Panic- 2019/10/08 05:20:35 http: panic serving 172.23.97.16:40064: runtime error: invalid memory address or nil pointer dereference goroutine 4088 [running]: net/http.(*conn).serve.func1(0xc00245aaa0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/server.go:1746 +0xd0 panic(0x1cfc080, 0x320d490) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/runtime/panic.go:513 +0x1b9 github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).doNameState(0xc0014500a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:304 +0x8db github.com/couchbase/query/clustering/couchbase.(*cbConfigStore).WhoAmI(0xc0014500a0, 0x0, 0x0, 0x0, 0xc000809fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/clustering/couchbase/clustering_cb.go:263 +0x2f github.com/couchbase/query/server/http.(*systemRemoteHttp).WhoAmI(0xc0011c5e60, 0x2147560, 0xc000809fb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/system_remote.go:566 +0x1e2 github.com/couchbase/query/prepareds.(*preparedCache).getPrepared(0x3427f90, 0x2147560, 0xc000809f60, 0x7, 0xc002753068, 0x37, 0x1c70ba0, 0xc000809f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:443 +0x5a1 github.com/couchbase/query/prepareds.GetPrepared(0x2147560, 0xc000809f60, 0xc000000007, 0xc002753068, 0xc000809ef0, 0x2147560, 0xc000809f60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/prepareds/prepareds.go:424 +0x53 github.com/couchbase/query/server/http.getPrepared(0x213c100, 0xc00231eae0, 0xc002753060, 0x8, 0x1c70ba0, 0xc000809ef0, 0xc002753068, 0xc0024cd9d8, 0x100000000ca804e, 0xc72591, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:620 +0x12f github.com/couchbase/query/server/http.handlePrepared(0xc001ca2000, 0x213c100, 0xc00231eae0, 0xc002753060, 0x8, 0x1c70ba0, 0xc000809ef0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:191 +0xa3 github.com/couchbase/query/server/http.(*jsonArgs).processParameters(0xc00231eae0, 0xc001ca2000, 0xd, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_request.go:1170 +0x158 ...   Logs- Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/322/ Logs- Panic in logs- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.15.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.20.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.16.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.17.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.23.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.12.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.13.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.14.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.177.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.19.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.20.zip  
          wayne Wayne Siu made changes -
          Field Original Value New Value
          Summary Panic in query logs for hight bucket density test [high-bucket] Panic in query logs for hight bucket density test
          raju Raju Suravarjjala made changes -
          Fix Version/s Mad-Hatter [ 15037 ]
          raju Raju Suravarjjala made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          keshav Keshav Murthy made changes -
          Assignee Keshav Murthy [ keshav ] Marco Greco [ marco.greco ]
          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - https://github.com/couchbase/query/blob/master/clustering/couchbase/clustering_cb.go#L299 typo check must be newErr
          marco.greco Marco Greco made changes -
          Affects Version/s 5.5.5 [ 16166 ]
          Affects Version/s 6.0.3 [ 16164 ]
          marco.greco Marco Greco added a comment -

          Issue has been there since Vulcan

          marco.greco Marco Greco added a comment - Issue has been there since Vulcan
          marco.greco Marco Greco made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]

          Build couchbase-server-6.5.0-4575 contains query commit e83eb7e with commit message:
          MB-36363 panic in clustering code

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4575 contains query commit e83eb7e with commit message: MB-36363 panic in clustering code

          Build couchbase-server-7.0.0-1012 contains query commit e83eb7e with commit message:
          MB-36363 panic in clustering code

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1012 contains query commit e83eb7e with commit message: MB-36363 panic in clustering code
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Marco Greco [ marco.greco ] Mahesh Mandhare [ mahesh.mandhare ]

          Build 6.5.0-4744

          Verified that panic is not happening following same steps as description.

          mahesh.mandhare Mahesh Mandhare (Inactive) added a comment - Build 6.5.0-4744 Verified that panic is not happening following same steps as description.
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            mahesh.mandhare Mahesh Mandhare (Inactive)
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty