Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32276

[System test]: Panic seen during longevity test

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 5.5.3
    • 5.5.3
    • eventing
    • None
    • centos1
    • Untriaged
    • Unknown

    Description

      build : 5.5.3 build 4037

      Eventing service panic during system test ,when we undeploy handler

      panic on node 172.23.96.56 

      [2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true
      [2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true 

      2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
      2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
      2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
      2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
      runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
      github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
              goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
      panic(0xc7fec0, 0xc432f5dc80)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
      github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
              goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
      created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
              goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
      2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
      2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
      2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
      2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          vikas.chaudhary Vikas Chaudhary created issue -
          vikas.chaudhary Vikas Chaudhary made changes -
          Field Original Value New Value
          Description build : 5.5.3 build 4037

          Eventing service panic during system test ,when we undeploy handler
          {noformat}
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
          2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
          runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
          panic(0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
          created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}
          build : 5.5.3 build 4037

          Eventing service panic during system test ,when we undeploy handler
          {noformat}
          [2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true
          [2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true {noformat}
          {noformat}2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
          2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
          runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
          panic(0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
          created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}
          vikas.chaudhary Vikas Chaudhary made changes -
          Description build : 5.5.3 build 4037

          Eventing service panic during system test ,when we undeploy handler
          {noformat}
          [2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true
          [2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true {noformat}
          {noformat}2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
          2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
          runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
          panic(0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
          created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}
          build : 5.5.3 build 4037

          Eventing service panic during system test ,when we undeploy handler

          panic on node 172.23.96.56 
          {noformat}[2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true
          [2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true {noformat}
          {noformat}2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
          2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
          2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
          runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
          panic(0xc7fec0, 0xc432f5dc80)
                  /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
          github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
          created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
                  goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
          2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}

          Multiple bugs this stacktrace have been raised in past on vulcan. In summary, this panic is harmless from functional PoV and it isn't like typical panics where processes are crashed.

          Fix for it has been pushed to Alice and porting it back to vulcan will too big of change at this point of the release cycle.

          asingh Abhishek Singh (Inactive) added a comment - Multiple bugs this stacktrace have been raised in past on vulcan. In summary, this panic is harmless from functional PoV and it isn't like typical panics where processes are crashed. Fix for it has been pushed to Alice and porting it back to vulcan will too big of change at this point of the release cycle.
          asingh Abhishek Singh (Inactive) made changes -
          Assignee Abhishek Singh [ asingh ] Vikas Chaudhary [ vikas.chaudhary ]

          closing as fixed in alice

          vikas.chaudhary Vikas Chaudhary added a comment - closing as fixed in alice
          vikas.chaudhary Vikas Chaudhary made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]
          vikas.chaudhary Vikas Chaudhary made changes -
          Actual End 2018-12-07 01:20 (issue has been closed)

          People

            vikas.chaudhary Vikas Chaudhary
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty