I root caused this MB to an issue with timers + rebalance out of an eventing node. The issue existed for some time now and got exposed in system longevity. We have a fix under Review & CI which should fix the problem.

Another thing we noticed is, when the eventing node is going out of the cluster, bucket ops through libcouchbase seem to slow down significantly. This part needs more investigation. Nonetheless, the patch should fix the problem of rebalance getting stuck

Fixes merged to mad-hatter branch. Testing done includes
http://qa.sc.couchbase.com/job/test_suite_executor/179827/testReport/eventing.eventing_rebalance/EventingRebalance/
http://qa.sc.couchbase.com/job/dev_testbed_blr1/159
CI Passed: http://ci-eventing.northscale.in/eventing-09.12.2019-15.29.pass.html

Build couchbase-server-6.5.0-4935 contains eventing commit 1e0312c with commit message:
MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

Build couchbase-server-7.0.0-1119 contains eventing commit 529bbc8 with commit message:
MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

Build couchbase-server-6.5.1-6005 contains eventing commit 1e0312c with commit message:
MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

Not seen on 6.5.0-4947