Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48572

[BP 7.0.2 MB-48570] - [Volume Test] Service 'eventing' exited with status 2.

    XMLWordPrintable

Details

    Description

      Build - 7.0.2 - 6683 (RC1)
      Test -

      -test tests/eventing/CC/test_eventing_volume.yml -scope tests/eventing/CC/scope_eventing_volume.yml
      

      Scale - 3

      On 172.23.104.23
      diag.log

      2021-09-23T00:22:31.777-07:00, ns_log:0:info:message(ns_1@172.23.104.23) - Service 'eventing' exited with status 2. Restarting. Messages:
      2021-09-23T00:22:31.763-07:00 [Info] Successfully stopped running HTTP server
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc05dd2]
       
      goroutine 85 [running]:
      net/http.(*Server).setupHTTP2_Serve(0x0, 0xc000142de0, 0xc0004d5c48)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:3145 +0x22
      net/http.(*Server).Serve(0x0, 0x1639000, 0xc0001a8d40, 0x0, 0x0)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2875 +0x14d
      github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4(0xc000124a00, 0x14a6302, 0x17, 0xc0004a2800, 0xc00011e4da, 0x6, 0xc00012a1c0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:428 +0x43d
      created by github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:388 +0x2c35
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jeelan.poola Jeelan Poola added a comment -

            Wayne SiuChanabasappa GhaliRitam Sharma We need to take the fix for this into 7.0.2. Request inclusion for RC2. Thank you!

            Abhishek Jindal Could you please add more details here on sequence of events that can lead to this crash? Thank you!

            jeelan.poola Jeelan Poola added a comment - Wayne Siu Chanabasappa Ghali Ritam Sharma We need to take the fix for this into 7.0.2. Request inclusion for RC2. Thank you! Abhishek Jindal Could you please add more details here on sequence of events that can lead to this crash? Thank you!

            It can happen while eventing service is starting up:

            • TLS server instance has been created but not yet Serve'ing any connections.
            • cbauth tls_cert change notification comes in which stops the existing server and sets the server instance to nil.
            • Existing server instance continues to serve the connections. However, this panics with nil ptr dereference because server instance was set to nil in Step 2 above.

            Fix is simple, we are now ensuring that a server instance gets set to nil sequentially in the same goroutine where it was initialised.

            abhishek.jindal Abhishek Jindal added a comment - It can happen while eventing service is starting up: TLS server instance has been created but not yet Serve'ing any connections. cbauth tls_cert change notification comes in which stops the existing server and sets the server instance to nil. Existing server instance continues to serve the connections. However, this panics with nil ptr dereference because server instance was set to nil in Step 2 above. Fix is simple, we are now ensuring that a server instance gets set to nil sequentially in the same goroutine where it was initialised.

            Build couchbase-server-7.0.2-6690 contains eventing commit 07a4ed1 with commit message:
            MB-48572 : Do not nullify server under stopserver

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6690 contains eventing commit 07a4ed1 with commit message: MB-48572 : Do not nullify server under stopserver
            sujay.gad Sujay Gad added a comment -

            Verified with volume test run on 7.0.2 - 6693.

            sujay.gad Sujay Gad added a comment - Verified with volume test run on 7.0.2 - 6693.

            People

              abhishek.jindal Abhishek Jindal
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty