Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37199

[System test]: Eventing rebalance hung since 13 hrs

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Build: 6.5.0-4926 , not seen on 6.5.0-4908

      Test: MH longevity

      Day: 2nd 

      Cycle: 2nd

      Rebalance out eventing node

      [2019-12-07T08:55:15-08:00, sequoiatools/couchbase-cli:6.5:699790] rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.148 -u Administrator -p password 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jeelan.poola Jeelan Poola added a comment -

          I root caused this MB to an issue with timers + rebalance out of an eventing node. The issue existed for some time now and got exposed in system longevity. We have a fix under Review & CI which should fix the problem.

          Another thing we noticed is, when the eventing node is going out of the cluster, bucket ops through libcouchbase seem to slow down significantly. This part needs more investigation. Nonetheless, the patch should fix the problem of rebalance getting stuck

          jeelan.poola Jeelan Poola added a comment - I root caused this MB to an issue with timers + rebalance out of an eventing node. The issue existed for some time now and got exposed in system longevity. We have a fix under Review & CI which should fix the problem. Another thing we noticed is, when the eventing node is going out of the cluster, bucket ops through libcouchbase seem to slow down significantly. This part needs more investigation. Nonetheless, the patch should fix the problem of rebalance getting stuck
          jeelan.poola Jeelan Poola added a comment - Fixes merged to mad-hatter branch. Testing done includes http://qa.sc.couchbase.com/job/test_suite_executor/179827/testReport/eventing.eventing_rebalance/EventingRebalance/ http://qa.sc.couchbase.com/job/dev_testbed_blr1/159 CI Passed: http://ci-eventing.northscale.in/eventing-09.12.2019-15.29.pass.html

          Build couchbase-server-6.5.0-4935 contains eventing commit 1e0312c with commit message:
          MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4935 contains eventing commit 1e0312c with commit message: MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          Build couchbase-server-7.0.0-1119 contains eventing commit 529bbc8 with commit message:
          MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1119 contains eventing commit 529bbc8 with commit message: MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          Build couchbase-server-6.5.1-6005 contains eventing commit 1e0312c with commit message:
          MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains eventing commit 1e0312c with commit message: MB-37199 : Stop generating oScanTimer internal events in c++ on an out going

          Not seen on 6.5.0-4947

          vikas.chaudhary Vikas Chaudhary added a comment - Not seen on 6.5.0-4947

          People

            jeelan.poola Jeelan Poola
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty