Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37199

[System test]: Eventing rebalance hung since 13 hrs

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Build: 6.5.0-4926 , not seen on 6.5.0-4908

      Test: MH longevity

      Day: 2nd 

      Cycle: 2nd

      Rebalance out eventing node

      [2019-12-07T08:55:15-08:00, sequoiatools/couchbase-cli:6.5:699790] rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.148 -u Administrator -p password

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Ascending order - Click to sort in descending order
          vikas.chaudhary Vikas Chaudhary created issue -
          jeelan.poola Jeelan Poola added a comment -

          I root caused this MB to an issue with timers + rebalance out of an eventing node. The issue existed for some time now and got exposed in system longevity. We have a fix under Review & CI which should fix the problem.

          Another thing we noticed is, when the eventing node is going out of the cluster, bucket ops through libcouchbase seem to slow down significantly. This part needs more investigation. Nonetheless, the patch should fix the problem of rebalance getting stuck

          jeelan.poola Jeelan Poola added a comment - I root caused this MB to an issue with timers + rebalance out of an eventing node. The issue existed for some time now and got exposed in system longevity. We have a fix under Review & CI which should fix the problem. Another thing we noticed is, when the eventing node is going out of the cluster, bucket ops through libcouchbase seem to slow down significantly. This part needs more investigation. Nonetheless, the patch should fix the problem of rebalance getting stuck
          lynn.straus Lynn Straus made changes -
          Field Original Value New Value
          Labels system-test approved-for-mad-hatter system-test
          lynn.straus Lynn Straus made changes -
          Due Date 10/Dec/19
          lynn.straus Lynn Straus made changes -
          Link This issue blocks MB-36676 [ MB-36676 ]
          jeelan.poola Jeelan Poola added a comment -

          Fixes merged to mad-hatter branch. Testing done includes
          http://qa.sc.couchbase.com/job/test_suite_executor/179827/testReport/eventing.eventing_rebalance/EventingRebalance/
          http://qa.sc.couchbase.com/job/dev_testbed_blr1/159
          CI Passed: http://ci-eventing.northscale.in/eventing-09.12.2019-15.29.pass.html

          jeelan.poola Jeelan Poola added a comment - Fixes merged to mad-hatter branch. Testing done includes http://qa.sc.couchbase.com/job/test_suite_executor/179827/testReport/eventing.eventing_rebalance/EventingRebalance/ http://qa.sc.couchbase.com/job/dev_testbed_blr1/159 CI Passed: http://ci-eventing.northscale.in/eventing-09.12.2019-15.29.pass.html
          jeelan.poola Jeelan Poola made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.5.0-4935 contains eventing commit 1e0312c with commit message:
          MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4935 contains eventing commit 1e0312c with commit message: MB-37199 : Stop generating oScanTimer internal events in c++ on an out going
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-1119 contains eventing commit 529bbc8 with commit message:
          MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1119 contains eventing commit 529bbc8 with commit message: MB-37199 : Stop generating oScanTimer internal events in c++ on an out going
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.5.1-6005 contains eventing commit 1e0312c with commit message:
          MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains eventing commit 1e0312c with commit message: MB-37199 : Stop generating oScanTimer internal events in c++ on an out going
          vikas.chaudhary Vikas Chaudhary added a comment -

          Not seen on 6.5.0-4947

          vikas.chaudhary Vikas Chaudhary added a comment - Not seen on 6.5.0-4947
          vikas.chaudhary Vikas Chaudhary made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            jeelan.poola Jeelan Poola
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty