[MB-32276] [System test]: Panic seen during longevity test - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 5.5.3
Fix Version/s: 5.5.3
Component/s: eventing
Labels:
None
Environment:
centos1

Triage:
Untriaged
Is this a Regression?:
Unknown

Description

build : 5.5.3 build 4037

Eventing service panic during system test ,when we undeploy handler

panic on node 172.23.96.56

[2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true

[2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true

2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer

2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine

2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine

2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:

runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)

        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79

github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)

        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e

panic(0xc7fec0, 0xc432f5dc80)

        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243

github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)

        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145

created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan

        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e

2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped

2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default

2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped

2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Vikas Chaudhary created issue - 06/Dec/18 10:46 PM

Vikas Chaudhary made changes - 06/Dec/18 10:47 PM

Field	Original Value	New Value
Description	build : 5.5.3 build 4037 Eventing service panic during system test ,when we undeploy handler {noformat} 2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer 2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine 2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine 2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]: runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79 github.com/couchbase/eventing/consumer.(Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00) goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e panic(0xc7fec0, 0xc432f5dc80) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243 github.com/couchbase/eventing/consumer.(Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0) goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145 created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e 2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped 2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default 2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped 2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}	build : 5.5.3 build 4037 Eventing service panic during system test ,when we undeploy handler {noformat} [2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true [2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true {noformat} {noformat}2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer 2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine 2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine 2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]: runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79 github.com/couchbase/eventing/consumer.(Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00) goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e panic(0xc7fec0, 0xc432f5dc80) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243 github.com/couchbase/eventing/consumer.(Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0) goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145 created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e 2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped 2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default 2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped 2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}

Vikas Chaudhary made changes - 06/Dec/18 10:48 PM

Description

build : 5.5.3 build 4037

Eventing service panic during system test ,when we undeploy handler
{noformat}
[2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true
[2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true {noformat}
{noformat}2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
panic(0xc7fec0, 0xc432f5dc80)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}

build : 5.5.3 build 4037

Eventing service panic during system test ,when we undeploy handler

panic on node 172.23.96.56
{noformat}[2018-12-04T00:45:09-08:00, sequoiatools/eventing:5.5:35940f] /eventing.py 172.23.96.56 8096 bucket_op_function_integration.json Administrator password undeploy true
[2018-12-04T00:45:27-08:00, sequoiatools/eventing:5.5:4ff5e0] /eventing.py 172.23.96.56 8096 bucket_op_complex_function_integration.json Administrator password undeploy true {noformat}
{noformat}2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Requested to stop supervisor for Eventing.Consumer
2018-12-04T00:45:12.673-08:00 [Info] Consumer::Stop [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting Consumer::Stop routine
2018-12-04T00:45:12.673-08:00 [Info] Consumer::handlerFailoverLog [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] Exiting failover log handling routine
2018-12-04T00:45:12.673-08:00 [Error] Consumer::addToAggChan [worker_bucket_op_function_0:/tmp/127.0.0.1:8091_worker_bucket_op_function_0.sock:1840] addToAggChan: recover send on closed channel stack trace: goroutine 1032299 [running]:
runtime/debug.Stack(0xc42ea83c98, 0xc7fec0, 0xc432f5dc80)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/debug/stack.go:24 +0x79
github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1.1(0xda11f7, 0x16, 0xc4250f8a00)
        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:775 +0x6e
panic(0xc7fec0, 0xc432f5dc80)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243
github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xda11f7, 0x16, 0xc4250f8a00, 0xc42d2994d0)
        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:808 +0x145
created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
        goproj/src/github.com/couchbase/eventing/consumer/process_events.go:812 +0x5e
2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38725:{eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
2018-12-04T00:45:12.681-08:00 [Info] Consumer::addToAggChan [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:1834] Closing dcp feed: {eventing:tbahHeMq-38720:172.23.96.56:8096_172.23.97.239:11210_worker_bucket_op_function_1}, count: 0 for bucket: default
2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] ##abcd ... stopped
2018-12-04T00:45:12.682-08:00 [Info] DCPT[eventing:tbahHeMq-38731:{eventing:tbahHeMq-38726:172.23.96.56:8096_172.23.99.20:11210_worker_bucket_op_function_1}/0] doReceive(): connection closed {noformat}

Abhishek Singh (Inactive) added a comment - 06/Dec/18 11:14 PM

Multiple bugs this stacktrace have been raised in past on vulcan. In summary, this panic is harmless from functional PoV and it isn't like typical panics where processes are crashed.

Fix for it has been pushed to Alice and porting it back to vulcan will too big of change at this point of the release cycle.

Abhishek Singh (Inactive) added a comment - 06/Dec/18 11:14 PM Multiple bugs this stacktrace have been raised in past on vulcan. In summary, this panic is harmless from functional PoV and it isn't like typical panics where processes are crashed. Fix for it has been pushed to Alice and porting it back to vulcan will too big of change at this point of the release cycle.

Abhishek Singh (Inactive) made changes - 06/Dec/18 11:15 PM

Assignee

Abhishek Singh [ asingh ]

Vikas Chaudhary [ vikas.chaudhary ]

Vikas Chaudhary added a comment - 06/Dec/18 11:20 PM

closing as fixed in alice

Vikas Chaudhary added a comment - 06/Dec/18 11:20 PM closing as fixed in alice

Vikas Chaudhary made changes - 06/Dec/18 11:20 PM

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Closed [ 6 ]

Vikas Chaudhary made changes - 06/Dec/18 11:20 PM

Actual End

2018-12-07 01:20 (issue has been closed)

People

Assignee:: Vikas Chaudhary

Reporter:: Vikas Chaudhary

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 06/Dec/18 10:46 PM

Updated:: 06/Dec/18 11:20 PM

Resolved:: 06/Dec/18 11:20 PM

Gerrit Reviews

There are no open Gerrit changes

PagerDuty

Couchbase Server

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty