Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.0
-
component cluster
-
Untriaged
-
-
Unknown
Description
Build : 6.0.0-1592
Test : -test tests/eventing/test_eventing_rebalance_timers.yml -scope tests/eventing/scope_eventing_rebalance_timers.yml
Scale : 2
Iteration : 2nd
During undeployment of a handler function, eventing service on 172.23.104.21 crashed and restarted.
The test was undeploying a handler function:
[2018-09-01T07:31:15-07:00, sequoiatools/eventing:6.0:9c0ebe] /eventing.py 172.23.104.19 8096 alice/timer_op.json Administrator password undeploy true
|
The following panic was seen in the eventing logs on 172.23.104.21 around the same time.
2018-09-01T07:31:15.397-07:00 [Info] DCPT[eventing:tbahHeMq-185:{eventing:tbahHeMq-184:172.23.104.21:8096_172.23.104.18:11210_worker_timer_op_complex_1}/0] DCP-socket -> eventing blocked 42.483923ms (0.993852%)
|
2018-09-01T07:31:15.490-07:00 [Info] SuperSupervisor::SettingsChangeCallback [3] Path => /eventing/appsettings/timer_op value => map[string]interface {}{"execution_timeout":60, "log_level":"INFO", "user_prefix":"eventing", "cleanup_timers":false, "deadline_timeout":62, "deployment_status":false, "worker_count":3, "dcp_stream_boundary":"from_now", "description":"", "processing_status":false}
|
2018-09-01T07:31:15.490-07:00 [Info] SuperSupervisor::SettingsChangeCallback [3] Function: timer_op current state: 2 requested status for deployment: false processing: false
|
2018-09-01T07:31:15.490-07:00 [Info] SuperSupervisor::SettingsChangeCallback [3] Function: timer_op Begin undeploy process. Current state: 2
|
2018-09-01T07:31:15.490-07:00 [Info] SuperSupervisor::SettingsChangeCallback [3] Function: timer_op enabled, settings change requesting undeployment
|
2018-09-01T07:31:15.490-07:00 [Info] SuperSupervisor::deleteFromLocallyDeployedApps [3] Function: timer_op deleting from locally deployed apps map
|
2018-09-01T07:31:15.490-07:00 [Info] SuperSupervisor::CleanupProducer [3] Function: timer_op stopping running instance of Eventing.Producer
|
2018-09-01T07:31:15.490-07:00 [Info] Producer::StopRunningConsumers [timer_op:3] Stopping running instances of Eventing.Consumer
|
2018-09-01T07:31:15.490-07:00 [Info] Consumer::Stop [worker_timer_op_1:/tmp/127.0.0.1:8091_worker_timer_op_1.sock:19009] Gracefully shutting down consumer routine
|
2018-09-01T07:31:15.491-07:00 [Info] Consumer::Stop [worker_timer_op_0:/tmp/127.0.0.1:8091_worker_timer_op_0.sock:19003] Gracefully shutting down consumer routine
|
2018-09-01T07:31:15.491-07:00 [Info] Consumer::Stop [worker_timer_op_2:/tmp/127.0.0.1:8091_worker_timer_op_2.sock:19008] Gracefully shutting down consumer routine
|
2018-09-01T07:31:15.491-07:00 [Info] Consumer::Stop [worker_timer_op_1:/tmp/127.0.0.1:8091_worker_timer_op_1.sock:19009] Issued close for go-couchbase and gocb handles
|
2018-09-01T07:31:15.491-07:00 [Info] Consumer::Stop [worker_timer_op_1:/tmp/127.0.0.1:8091_worker_timer_op_1.sock:19009] Requested to remove supervision of eventing-consumer
|
2018-09-01T07:31:15.491-07:00 [Info] Consumer::Stop [worker_timer_op_1:/tmp/127.0.0.1:8091_worker_timer_op_1.sock:19009] Stopped checkpoint, restart vb dcp stream and stats tickers
|
2018-09-01T07:31:15.491-07:00 [Info] Consumer::sendMessageLoop [worker_timer_op_1:/tmp/127.0.0.1:8091_worker_timer_op_1.sock:19009] Exiting send message routine
|
panic: send on closed channel
|
|
goroutine 2332 [running]:
|
panic(0xbeee00, 0xdbb7b0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/panic.go:551 +0x3c1 fp=0xc429c44ca0 sp=0xc429c44c00 pc=0x42e191
|
runtime.chansend(0xc426112f60, 0xc4294d2578, 0xc4294d2501, 0xa09a07, 0xc421b45378)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/chan.go:245 +0x583 fp=0xc429c44d28 sp=0xc429c44ca0 pc=0x407493
|
runtime.chansend1(0xc426112f60, 0xc4294d2578)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/chan.go:125 +0x35 fp=0xc429c44d60 sp=0xc429c44d28 pc=0x406f05
|
github.com/couchbase/eventing/util.(*BoundedQueue).Push(0xc425fdeb40, 0xdbc660, 0xc429b5d450, 0x0, 0xc421b45378)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/bounded_queue.go:49 +0xf7 fp=0xc429c44d98 sp=0xc429c44d60 pc=0xa09a07
|
github.com/couchbase/eventing/consumer.(*Consumer).routeTimers.func1(0xc4223ab200, 0x1, 0xc429b5d450, 0xc429c44f08, 0xd0e745, 0x15)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/timer.go:154 +0xd7 fp=0xc429c44e58 sp=0xc429c44d98 pc=0xaac1e7
|
github.com/couchbase/eventing/consumer.(*Consumer).routeTimers(0xc4223ab200)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/timer.go:159 +0x330 fp=0xc429c44fd8 sp=0xc429c44e58 pc=0xa7fe60
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc429c44fe0 sp=0xc429c44fd8 pc=0x45db11
|
created by github.com/couchbase/eventing/consumer.(*Consumer).HandleV8Worker
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:349 +0x4e9
|
|
goroutine 1 [chan receive, 7 minutes]:
|
runtime.gopark(0xd3ca98, 0xc4201b17f8, 0xd078e7, 0xc, 0x217, 0x3)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:291 +0x11a fp=0xc42045d580 sp=0xc42045d560 pc=0x4302fa
|
runtime.goparkunlock(0xc4201b17f8, 0xd078e7, 0xc, 0xc4200c4017, 0x3)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:297 +0x5e fp=0xc42045d5c0 sp=0xc42045d580 pc=0x4303ae
|
runtime.chanrecv(0xc4201b17a0, 0xc420191a60, 0xd3cc01, 0xc4201916b8)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/chan.go:518 +0x2f2 fp=0xc42045d658 sp=0xc42045d5c0 pc=0x407dc2
|
runtime.chanrecv1(0xc4201b17a0, 0xc420191a60)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/chan.go:400 +0x2b fp=0xc42045d688 sp=0xc42045d658 pc=0x407a7b
|
github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg(0xc420474fc0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:666 +0x73 fp=0xc42045de48 sp=0xc42045d688 pc=0xb1cb93
|
main.main()
|
goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 +0x3fc fp=0xc42045df88 sp=0xc42045de48 pc=0xb2726c
|
Error on UI Diag logs
Service 'eventing' exited with status 134. Restarting. Messages:
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/chan.go:234 +0x25b fp=0xc42e0fc5a0 sp=0xc42e0fc518 pc=0x40716b
|
runtime.chansend1(0xc42d704ae0, 0xc42e0fc640)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/chan.go:125 +0x35 fp=0xc42e0fc5d8 sp=0xc42e0fc5a0 pc=0x406f05
|
github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1(0xd0f3f5, 0x16, 0xc42c5c3200, 0xc43015b950)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:729 +0x111 fp=0xc42e0fc7c0 sp=0xc42e0fc5d8 pc=0xaaad81
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42e0fc7c8 sp=0xc42e0fc7c0 pc=0x45db11
|
created by github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:687 +0x5e
|
[goport(/opt/couchbase/bin/eventing-producer)] 2018/09/01 07:31:16 child process exited with status 134
|