Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42473

[System Test] Panic observed in query logs

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-3583
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 5
      Day : 4th

      In the system test, after the 5th iteration started, the query workload after recreating the indexes was this - 

      [2020-11-02T23:13:27-08:00, sequoiatools/cbdozer:b0dac7] -method POST -duration 0 -rate 10 -url http://Administrator:password@172.23.96.251:8093:8095/query/service -body delete from default where rating > 0 limit 30

      The following stack trace was seen on one of the query nodes 172.23.97.149:

      2020-11-02T23:15:20.967-08:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
      		 
      2020-11-02T23:15:20.967-08:00 [Info] Refreshed Indexer List: [172.23.123.32:9100 172.23.96.243:9100 172.23.96.254:9100 172.23.97.105:9100 172.23.97.110:9100 172.23.97.112:9100]
      		 
      2020-11-02T23:15:20.976-08:00 [Info] switched currmeta from 3661 -> 3661 force true
      		 
      2020-11-02T23:15:20.977-08:00 [Info] serviceChangeNotifier: received CollectionManifestChangeNotification for bucket: bucket8
      		 
      _time=2020-11-02T23:15:21.140-08:00 _level=SEVERE _msg= panic=runtime error: invalid memory address or nil pointer dereference stack=goroutine 11588623 [running]:
      		 
      github.com/couchbase/query/server.(*Server).serviceRequest.func1()
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:865 +0xb8
      		 
      panic(0x21af4e0, 0x3830620)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/query/execution.OpStop(...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:517
      		 
      github.com/couchbase/query/server.(*BaseRequest).Stop(0xc005926000, 0xc000000004)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/request.go:1007 +0x9f
      		 
      github.com/couchbase/query/server/http.(*httpRequest).Execute(0xc005926000, 0xc00000c3c0, 0xc0026198c0, 0x249398c, 0xb, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_response.go:144 +0x44f
      		 
      github.com/couchbase/query/server.(*Server).serviceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:983 +0x8f5
      		 
      github.com/couchbase/query/server.(*Server).handleRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0xc00000c3f0, 0x7)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:587 +0x6c
      		 
      github.com/couchbase/query/server.(*Server).ServiceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:528 +0x77
      		 
      github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc00039d4a0, 0x27401e0, 0xc007273c00, 0xc005fc1a00)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:251 +0x35d
      		 
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc0009e8180, 0x27401e0, 0xc007273c00, 0xc005fc1300)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
      		 
      net/http.serverHandler.ServeHTTP(0xc00003a000, 0x27401e0, 0xc007273c00, 0xc005fc1300)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
      		 
      net/http.(*conn).serve(0xc003b78640, 0x274bd20, 0xc005fab280)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
      		 
      created by net/http.(*Server).Serve
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
      		 
       
      		 
      goroutine 11588623 [running]:
      		 
      github.com/couchbase/query/server.(*Server).serviceRequest.func1()
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:865 +0xb8
      		 
      panic(0x21af4e0, 0x3830620)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/query/execution.OpStop(...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:517
      		 
      github.com/couchbase/query/server.(*BaseRequest).Stop(0xc005926000, 0xc000000004)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/request.go:1007 +0x9f
      		 
      github.com/couchbase/query/server/http.(*httpRequest).Execute(0xc005926000, 0xc00000c3c0, 0xc0026198c0, 0x249398c, 0xb, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_response.go:144 +0x44f
      		 
      github.com/couchbase/query/server.(*Server).serviceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:983 +0x8f5
      		 
      github.com/couchbase/query/server.(*Server).handleRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0xc00000c3f0, 0x7)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:587 +0x6c
      		 
      github.com/couchbase/query/server.(*Server).ServiceRequest(0xc00000c3c0, 0x27be8e0, 0xc005926000, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:528 +0x77
      		 
      github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP(0xc00039d4a0, 0x27401e0, 0xc007273c00, 0xc005fc1a00)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:251 +0x35d
      		 
      github.com/gorilla/mux.(*Router).ServeHTTP(0xc0009e8180, 0x27401e0, 0xc007273c00, 0xc005fc1300)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
      		 
      net/http.serverHandler.ServeHTTP(0xc00003a000, 0x27401e0, 0xc007273c00, 0xc005fc1300)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
      		 
      net/http.(*conn).serve(0xc003b78640, 0x274bd20, 0xc005fab280)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
      		 
      created by net/http.(*Server).Serve
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
      		 
      _time=2020-11-02T23:15:21.572-08:00 _level=INFO _msg= Retrying bucket N1QL_SYSTEM_BUCKET
      		 
      _time=2020-11-02T23:15:21.582-08:00 _level=INFO _msg= keyspace N1QL_SYSTEM_BUCKET not found No bucket named N1QL_SYSTEM_BUCKET

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty