Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36972

Undeployment fails with ERR_REBALANCE_ONGOING eventhough rebalance completed

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Build: 6.5.0-4821 , last time passed on 6.5.0-4783

      ./testrunner -i /tmp/testexec.7233.ini -p get-cbcollect-info=True,GROUP=bucket_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_eventing_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-eventing-eventing-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers
      
      

      Observed following rebalance completed 

      [2019-11-15 08:04:13,709] - [rest_client:2105] INFO - Node 172.23.105.80 not part of cluster inactiveAdded
      [2019-11-15 08:04:14,628] - [rest_client:1497] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.94,ns_1@172.23.105.96,ns_1@172.23.105.59', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.32,ns_1@172.23.105.80,ns_1@172.23.105.82,ns_1@172.23.105.79,ns_1@172.23.105.96,ns_1@172.23.105.59,ns_1@172.23.105.94'}
      [2019-11-15 08:04:14,642] - [rest_client:1502] INFO - rebalance operation started
      [2019-11-15 08:04:14,649] - [rest_client:1659] INFO - rebalance percentage : 0.00 %
      [2019-11-15 08:04:14,650] - [task:533] INFO - Rebalance - status: running, progress: 0.00%
      [2019-11-15 08:04:16,761] - [rest_client:92] INFO - rebalance reached >100% in 3.00745296478 seconds 
      [2019-11-15 08:04:24,682] - [rest_client:1659] INFO - rebalance percentage : 21.43 %
      [2019-11-15 08:04:24,683] - [task:533] INFO - Rebalance - status: running, progress: 21.43%
      [2019-11-15 08:04:34,708] - [rest_client:1659] INFO - rebalance percentage : 36.29 %
      [2019-11-15 08:04:34,708] - [task:533] INFO - Rebalance - status: running, progress: 36.29%
      [2019-11-15 08:04:44,733] - [rest_client:1659] INFO - rebalance percentage : 65.81 %
      [2019-11-15 08:04:44,733] - [task:533] INFO - Rebalance - status: running, progress: 65.81%
      [2019-11-15 08:04:54,761] - [rest_client:1659] INFO - rebalance percentage : 91.23 %
      [2019-11-15 08:04:54,761] - [task:533] INFO - Rebalance - status: running, progress: 91.23%
      [2019-11-15 08:05:04,788] - [task:533] INFO - Rebalance - status: none, progress: 100.00%
      [2019-11-15 08:05:04,844] - [task:592] INFO - rebalancing was completed with progress: 100% in 50.2019569874 sec
      
      

      But undeployment fails with 

      [2019-11-15 08:05:12,632] - [rest_client:840] ERROR - POST http://172.23.105.80:8096/api/v1/functions/Function_771968608_test_eventing_rebalance_with_multiple_eventing_nodes/settings body: {"processing_status": false, "deployment_status": false} headers: {'Content-type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 406 reason: unknown {
       "name": "ERR_REBALANCE_ONGOING",
       "code": 36,
       "description": "Rebalance ongoing on some/all Eventing nodes, creating new functions, deployment or undeployment of existing functions is not allowed",
       "attributes": null,
       "runtime_info": {
        "code": 36,
        "info": "Rebalance ongoing on some/all Eventing nodes, creating new functions, deployment or undeployment of existing functions is not allowed"
       }
      } auth: Administrator:password
      ERROR 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ankit.prabhu Ankit Prabhu
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty