Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
6.5.0
-
Untriaged
-
Yes
Description
Build: 6.5.0-4926 , not seen on 6.5.0-4908
Test: MH longevity
Day: 2nd
Cycle: 2nd
Rebalance out eventing node
[2019-12-07T08:55:15-08:00, sequoiatools/couchbase-cli:6.5:699790] rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.148 -u Administrator -p password
|
Attachments
| For Gerrit Dashboard: MB-37199 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 119059,3 | MB-37199 : Stop generating oScanTimer internal events in c++ on an out going eventing node and during pause. Separate out Add/Remove Timer Partitions | unstable | eventing | Status: MERGED | +2 | +1 |
| 119123,3 | MB-37199 : Stop generating oScanTimer internal events in c++ on an out going eventing node and during pause. Separate out Add/Remove Timer Partitions | mad-hatter | eventing | Status: MERGED | +2 | +1 |
Activity
| Field | Original Value | New Value |
|---|---|---|
| Labels | system-test | approved-for-mad-hatter system-test |
| Due Date | 10/Dec/19 |
| Link | This issue blocks MB-36676 [ MB-36676 ] |
Fixes merged to mad-hatter branch. Testing done includes
http://qa.sc.couchbase.com/job/test_suite_executor/179827/testReport/eventing.eventing_rebalance/EventingRebalance/
http://qa.sc.couchbase.com/job/dev_testbed_blr1/159
CI Passed: http://ci-eventing.northscale.in/eventing-09.12.2019-15.29.pass.html
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
Build couchbase-server-6.5.0-4935 contains eventing commit 1e0312c with commit message:
MB-37199 : Stop generating oScanTimer internal events in c++ on an out going
Build couchbase-server-7.0.0-1119 contains eventing commit 529bbc8 with commit message:
MB-37199 : Stop generating oScanTimer internal events in c++ on an out going
Build couchbase-server-6.5.1-6005 contains eventing commit 1e0312c with commit message:
MB-37199 : Stop generating oScanTimer internal events in c++ on an out going
| Status | Resolved [ 5 ] | Closed [ 6 ] |
I root caused this MB to an issue with timers + rebalance out of an eventing node. The issue existed for some time now and got exposed in system longevity. We have a fix under Review & CI which should fix the problem.
Another thing we noticed is, when the eventing node is going out of the cluster, bucket ops through libcouchbase seem to slow down significantly. This part needs more investigation. Nonetheless, the patch should fix the problem of rebalance getting stuck