Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36839

Multiple KV rebalance failed with Some apps are deploying

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Build: 6.5.0-4783 not seen on 4744

      While rebalancing multiple kv node 

      [2019-11-10 02:13:26,527] - [rest_client:1497] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.104.80,ns_1@172.23.104.199', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.86,ns_1@172.23.104.199,ns_1@172.23.105.109,ns_1@172.23.104.97,ns_1@172.23.104.80,ns_1@172.23.105.44,ns_1@172.23.105.114'}
      

       2019-11-10 02:14:47,129] - [rest_client:3325] ERROR - {u'node': u'ns_1@172.23.104.199', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.26798.8>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 5b71a224604a54e463dc174951e48c2f', u'shortText': u'message', u'serverTime': u'2019-11-10T02:14:42.267Z', u'module': u'ns_orchestrator', u'tstamp': 1573380882267, u'type': u'critical'}

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Hey Sergey Avseyev,

            I ran the test twice now and once with your previous patch, it ran successfully every time. I observed there were no crashes and it was a clean run. With that being said, the issue was intermittent and the repro was not always seen so cannot confirm whether is as gone away fully.

            Thanks.

            suraj.naik Suraj Naik (Inactive) added a comment - Hey Sergey Avseyev , I ran the test twice now and once with your previous patch, it ran successfully every time. I observed there were no crashes and it was a clean run. With that being said, the issue was intermittent and the repro was not always seen so cannot confirm whether is as gone away fully. Thanks.

            The patch has been merged

            avsej Sergey Avseyev added a comment - The patch has been merged

            The underlying issue is fixed, please update the dependency accordingly and then we should see this fixed as well.

            We may want to update it one more time to pick up any other fixes in the December maintenance release, as that goes through a more thorough QE cycle and picks up other possible fixes. That's targeted for December 3.

            ingenthr Matt Ingenthron added a comment - The underlying issue is fixed, please update the dependency accordingly and then we should see this fixed as well. We may want to update it one more time to pick up any other fixes in the December maintenance release, as that goes through a more thorough QE cycle and picks up other possible fixes. That's targeted for December 3.

            Fix is available from build couchbase-server-6.5.0-4878 onwards.

            jeelan.poola Jeelan Poola added a comment - Fix is available from build couchbase-server-6.5.0-4878 onwards.

            Not seeing on 6.5.0-4908
             

            vikas.chaudhary Vikas Chaudhary added a comment - Not seeing on 6.5.0-4908  

            People

              jeelan.poola Jeelan Poola
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty