Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42473

[System Test] Panic observed in query logs

    XMLWordPrintable

    Details

      Description

      Build : 7.0.0-3583
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 5
      Day : 4th

      In the system test, after the 5th iteration started, the query workload after recreating the indexes was this -

      [2020-11-02T23:13:27-08:00, sequoiatools/cbdozer:b0dac7] -method POST -duration 0 -rate 10 -url http://Administrator:password@172.23.96.251:8093:8095/query/service -body delete from default where rating > 0 limit 30
      

      The following stack trace was seen on one of the query nodes 172.23.97.149:

      2020-11-02T23:15:20.967-08:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
      2020-11-02T23:15:20.967-08:00 [Info] Refreshed Indexer List: [172.23.123.32:9100 172.23.96.243:9100 172.23.96.254:9100 172.23.97.105:9100 172.23.97.110:9100 172.23.97.112:9100]
      2020-11-02T23:15:20.976-08:00 [Info] switched currmeta from 3661 -> 3661 force true
      2020-11-02T23:15:20.977-08:00 [Info] serviceChangeNotifier: received CollectionManifestChangeNotification for bucket: bucket8
      _time=2020-11-02T23:15:21.140-08:00 _level=SEVERE _msg= panic=runtime error: invalid memory address or nil pointer dereference stack=goroutine 11588623 [running]:
      github.com/couchbase/query/server.(*Server).serviceRequest.func1()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:865 +0xb8
      panic(0x21af4e0, 0x3830620)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      github.com/couchbase/query/execution.OpStop(...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:517
      github.com/couchbase/query/server.(*BaseRequest).Stop(0xc005926000, 0xc000000004)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/request.go:1007 +0x9f
      github.com/couchbase/query/server/http.(*httpRequest).Execute(0xc005926000, 0xc00000c3c0, 0xc0026198c0, 0x249398c, 0xb, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_response.go:144 +0x44f
      github.com/couchbase/query/server.(*Server).serviceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:983 +0x8f5
      github.com/couchbase/query/server.(*Server).handleRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0xc00000c3f0, 0x7)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:587 +0x6c
      github.com/couchbase/query/server.(*Server).ServiceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:528 +0x77
      github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc00039d4a0, 0x27401e0, 0xc007273c00, 0xc005fc1a00)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:251 +0x35d
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc0009e8180, 0x27401e0, 0xc007273c00, 0xc005fc1300)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
      net/http.serverHandler.ServeHTTP(0xc00003a000, 0x27401e0, 0xc007273c00, 0xc005fc1300)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
      net/http.(*conn).serve(0xc003b78640, 0x274bd20, 0xc005fab280)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
      created by net/http.(*Server).Serve
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
       
      goroutine 11588623 [running]:
      github.com/couchbase/query/server.(*Server).serviceRequest.func1()
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:865 +0xb8
      panic(0x21af4e0, 0x3830620)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      github.com/couchbase/query/execution.OpStop(...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:517
      github.com/couchbase/query/server.(*BaseRequest).Stop(0xc005926000, 0xc000000004)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/request.go:1007 +0x9f
      github.com/couchbase/query/server/http.(*httpRequest).Execute(0xc005926000, 0xc00000c3c0, 0xc0026198c0, 0x249398c, 0xb, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_response.go:144 +0x44f
      github.com/couchbase/query/server.(*Server).serviceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:983 +0x8f5
      github.com/couchbase/query/server.(*Server).handleRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0xc00000c3f0, 0x7)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:587 +0x6c
      github.com/couchbase/query/server.(*Server).ServiceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:528 +0x77
      github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc00039d4a0, 0x27401e0, 0xc007273c00, 0xc005fc1a00)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:251 +0x35d
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc0009e8180, 0x27401e0, 0xc007273c00, 0xc005fc1300)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
      net/http.serverHandler.ServeHTTP(0xc00003a000, 0x27401e0, 0xc007273c00, 0xc005fc1300)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
      net/http.(*conn).serve(0xc003b78640, 0x274bd20, 0xc005fab280)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
      created by net/http.(*Server).Serve
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
      _time=2020-11-02T23:15:21.572-08:00 _level=INFO _msg= Retrying bucket N1QL_SYSTEM_BUCKET
      _time=2020-11-02T23:15:21.582-08:00 _level=INFO _msg= keyspace N1QL_SYSTEM_BUCKET not found No bucket named N1QL_SYSTEM_BUCKET
      

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          marco.greco Marco Greco added a comment -

          This is a timeout and a client close arriving at the same time. the first goes through and nillifies this.stopOperator for the second

          Show
          marco.greco Marco Greco added a comment - This is a timeout and a client close arriving at the same time. the first goes through and nillifies this.stopOperator for the second
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-3631 contains query commit 6449990 with commit message:
          MB-42473 race condition betwen timeout and close

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3631 contains query commit 6449990 with commit message: MB-42473 race condition betwen timeout and close
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.1-9170 contains query commit 9331b16 with commit message:
          MB-42473 race condition betwen timeout and close

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9170 contains query commit 9331b16 with commit message: MB-42473 race condition betwen timeout and close
          Hide
          mihir.kamdar Mihir Kamdar added a comment -

          Closing this as this issue was not seen in a 5 day run of the same test with 6.6.1-9207

          Show
          mihir.kamdar Mihir Kamdar added a comment - Closing this as this issue was not seen in a 5 day run of the same test with 6.6.1-9207

            People

            Assignee:
            mihir.kamdar Mihir Kamdar
            Reporter:
            mihir.kamdar Mihir Kamdar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty