Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44792

[REBALANCE] during upgrade tests rebalance fails/stops

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      During the query upgrade regression tests we see many rebalance errors such as :

      test_6:

      membase.api.exception.RebalanceFailedException: Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

      test_7:

      membase.api.exception.RebalanceFailedException: Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance stopped by janitor.'} - rebalance failed ERROR

      test_11:

      membase.api.exception.RebalanceFailedException: Rebalance Failed: seems like rebalance hangs. please check logs!

      This happens when nodes are removed and rebalance is invoked:

      Flow in test_6 or test_7:

      [2021-03-04 14:01:19,009] - [newupgradebasetest] [139890759030528] - INFO - upgrade 172.23.100.11 to version 7.0.0-4554 is completed
      		 
      [2021-03-04 14:01:19,009] - [basetestcase] [139890759030528] - INFO - sleep for 30 secs.  ...
      		 
      [2021-03-04 14:01:49,046] - [cluster_helper] [139890759030528] - INFO - waiting for ns_server @ 172.23.100.11:8091
      		 
      [2021-03-04 14:01:49,046] - [rest_client] [139890759030528] - INFO - -->is_ns_server_running?
      		 
      [2021-03-04 14:01:49,053] - [cluster_helper] [139890759030528] - INFO - ns_server @ 172.23.100.11:8091 is running
      		 
      [2021-03-04 14:01:49,053] - [rest_client] [139890759030528] - INFO - --> in init_cluster...Administrator,password,8091
      		 
      [2021-03-04 14:01:49,053] - [rest_client] [139890759030528] - INFO - settings/web params on 172.23.100.11:8091:port=8091&username=Administrator&password=password
      		 
      [2021-03-04 14:01:49,108] - [rest_client] [139890759030528] - INFO - --> status:True
      		 
      [2021-03-04 14:01:49,108] - [basetestcase] [139890759030528] - INFO - sleep for 15 secs.  ...
      		 
      [2021-03-04 14:02:04,123] - [basetestcase] [139890759030528] - INFO - sleep for 30 secs.  ...
      		 
      [2021-03-04 14:02:34,956] - [task] [139890767423232] - INFO - adding node 172.23.100.11:8091 to cluster
      		 
      [2021-03-04 14:02:34,956] - [rest_client] [139890767423232] - INFO - adding remote node @172.23.100.11:8091 to this cluster @172.23.100.15:8091
      		 
      [2021-03-04 14:02:34,960] - [rest_client] [139890767423232] - ERROR - {'status': 'none', 'errorMessage': 'Rebalance stopped by janitor.'} - rebalance failed
      		 
      [2021-03-04 14:02:34,981] - [rest_client] [139890767423232] - INFO - Latest logs from UI on 172.23.100.15:
      		 
      [2021-03-04 14:02:34,982] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1614895290170, 'shortText': 'message', 'text': 'Enabled auto-failover with timeout 120 and max count 1', 'serverTime': '2021-03-04T14:01:30.170Z'}
      		 
      [2021-03-04 14:02:34,982] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'mb_master', 'tstamp': 1614895290137, 'shortText': 'message', 'text': "Haven't heard from a higher priority node or a master, so I'm taking over.", 'serverTime': '2021-03-04T14:01:30.137Z'}
      		 
      [2021-03-04 14:02:34,982] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1614895281140, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.100.11\' in 0 seconds.', 'serverTime': '2021-03-04T14:01:21.140Z'}
      		 
      [2021-03-04 14:02:34,983] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'warning', 'code': 0, 'module': 'mb_master', 'tstamp': 1614895280116, 'shortText': 'message', 'text': "Current master is older and I'll try to takeover", 'serverTime': '2021-03-04T14:01:20.116Z'}
      		 
      [2021-03-04 14:02:34,983] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1614895279947, 'shortText': 'message', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.100.11\'. Version: "7.0.0-4554-enterprise".', 'serverTime': '2021-03-04T14:01:19.947Z'}
      		 
      [2021-03-04 14:02:34,983] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1614895279852, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.100.11' saw that node 'ns_1@172.23.100.15' came up. Tags: []", 'serverTime': '2021-03-04T14:01:19.852Z'}
      		 
      [2021-03-04 14:02:34,983] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.15', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1614895279850, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.100.15' saw that node 'ns_1@172.23.100.11' came up. Tags: []", 'serverTime': '2021-03-04T14:01:19.850Z'}
      		 
      [2021-03-04 14:02:34,984] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1614895279833, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.100.11' saw that node 'ns_1@172.23.100.14' came up. Tags: []", 'serverTime': '2021-03-04T14:01:19.833Z'}
      		 
      [2021-03-04 14:02:34,984] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.14', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1614895279830, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.100.14' saw that node 'ns_1@172.23.100.11' came up. Tags: []", 'serverTime': '2021-03-04T14:01:19.830Z'}
      		 
      [2021-03-04 14:02:34,984] - [rest_client] [139890767423232] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1614895279806, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.100.11' saw that node 'ns_1@172.23.100.10' came up. Tags: []", 'serverTime': '2021-03-04T14:01:19.806Z'}

       

      Flow in test_11:

      [2021-03-04 15:55:21,246] - [newupgradebasetest] [139888921913088] - INFO - upgrade 172.23.100.15 to version 7.0.0-4554 is completed
      		 
      [2021-03-04 15:55:21,247] - [basetestcase] [139888921913088] - INFO - sleep for 30 secs.  ...
      		 
      [2021-03-04 15:55:51,273] - [cluster_helper] [139888921913088] - INFO - waiting for ns_server @ 172.23.100.15:8091
      		 
      [2021-03-04 15:55:51,273] - [rest_client] [139888921913088] - INFO - -->is_ns_server_running?
      		 
      [2021-03-04 15:55:51,278] - [cluster_helper] [139888921913088] - INFO - ns_server @ 172.23.100.15:8091 is running
      		 
      [2021-03-04 15:55:51,278] - [rest_client] [139888921913088] - INFO - --> in init_cluster...Administrator,password,8091
      		 
      [2021-03-04 15:55:51,278] - [rest_client] [139888921913088] - INFO - settings/web params on 172.23.100.15:8091:port=8091&username=Administrator&password=password
      		 
      [2021-03-04 15:55:51,346] - [rest_client] [139888921913088] - INFO - --> status:True
      		 
      [2021-03-04 15:55:51,347] - [basetestcase] [139888921913088] - INFO - sleep for 15 secs.  ...
      		 
      [2021-03-04 15:56:06,352] - [basetestcase] [139888921913088] - INFO - sleep for 30 secs.  ...
      		 
      [2021-03-04 15:56:36,978] - [rest_client] [139888393434880] - INFO - Node versions in cluster ['6.5.1-6299-enterprise', '7.0.0-4554-enterprise', '6.5.1-6299-enterprise']
      		 
      [2021-03-04 15:56:37,296] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:56:37,296] - [task] [139888393434880] - INFO - adding node 172.23.100.15:8091 to cluster
      		 
      [2021-03-04 15:56:37,296] - [rest_client] [139888393434880] - INFO - adding remote node @172.23.100.15:8091 to this cluster @172.23.100.11:8091
      		 
      [2021-03-04 15:56:47,313] - [rest_client] [139888393434880] - INFO - rebalance progress took 10.02 seconds 
      [2021-03-04 15:56:47,313] - [rest_client] [139888393434880] - INFO - sleep for 10 seconds after rebalance...
      		 
      [2021-03-04 15:57:01,064] - [rest_client] [139888393434880] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.10,ns_1@172.23.100.11,ns_1@172.23.100.14,ns_1@172.23.100.15', 'ejectedNodes': 'ns_1@172.23.100.14', 'user': 'Administrator', 'password': 'password'}
      		 
      [2021-03-04 15:57:01,077] - [rest_client] [139888393434880] - INFO - rebalance operation started
      		 
      [2021-03-04 15:57:01,078] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:57:01,825] - [rest_client] [139888393434880] - INFO - rebalance percentage : 0.00 %
      		 
      [2021-03-04 15:57:01,825] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 0.00%
      		 
      [2021-03-04 15:57:11,856] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:57:14,204] - [rest_client] [139888393434880] - INFO - rebalance percentage : 0.00 %
      		 
      [2021-03-04 15:57:14,204] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 0.00%
      		 
      [2021-03-04 15:57:27,066] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:57:30,013] - [rest_client] [139888393434880] - INFO - rebalance percentage : 1.00 %
      		 
      [2021-03-04 15:57:30,013] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 1.00%
      		 
      [2021-03-04 15:57:40,678] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:57:43,939] - [rest_client] [139888393434880] - INFO - rebalance percentage : 2.00 %
      		 
      [2021-03-04 15:57:43,939] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 2.00%
      		 
      [2021-03-04 15:57:55,195] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:57:57,224] - [rest_client] [139888393434880] - INFO - rebalance percentage : 3.00 %
      		 
      [2021-03-04 15:57:57,224] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 3.00%
      		 
      [2021-03-04 15:58:07,366] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:58:09,829] - [rest_client] [139888393434880] - INFO - rebalance percentage : 4.00 %
      		 
      [2021-03-04 15:58:09,830] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 4.00%
      		 
      [2021-03-04 15:58:21,683] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:58:24,032] - [rest_client] [139888393434880] - INFO - rebalance percentage : 5.00 %
      		 
      [2021-03-04 15:58:24,032] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 5.00%
      		 
      [2021-03-04 15:58:35,434] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:58:38,408] - [rest_client] [139888393434880] - INFO - rebalance percentage : 6.00 %
      		 
      [2021-03-04 15:58:38,409] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 6.00%
      		 
      [2021-03-04 15:58:49,833] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:58:51,665] - [rest_client] [139888393434880] - INFO - rebalance percentage : 7.00 %
      		 
      [2021-03-04 15:58:51,666] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 7.00%
      		 
      [2021-03-04 15:59:01,774] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:59:03,723] - [rest_client] [139888393434880] - INFO - rebalance percentage : 8.00 %
      		 
      [2021-03-04 15:59:03,724] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 8.00%
      		 
      [2021-03-04 15:59:14,241] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:59:16,224] - [rest_client] [139888393434880] - INFO - rebalance percentage : 9.00 %
      		 
      [2021-03-04 15:59:16,224] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 9.00%
      		 
      [2021-03-04 15:59:26,477] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:59:28,507] - [rest_client] [139888393434880] - INFO - rebalance percentage : 11.00 %
      		 
      [2021-03-04 15:59:28,507] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 11.00%
      		 
      [2021-03-04 15:59:38,659] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:59:42,671] - [rest_client] [139888393434880] - INFO - rebalance percentage : 11.00 %
      		 
      [2021-03-04 15:59:42,671] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 11.00%
      		 
      [2021-03-04 15:59:54,398] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 15:59:56,811] - [rest_client] [139888393434880] - INFO - rebalance percentage : 12.00 %
      		 
      [2021-03-04 15:59:56,811] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 12.00%
      		 
      [2021-03-04 16:00:08,022] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:00:11,554] - [rest_client] [139888393434880] - INFO - rebalance percentage : 13.00 %
      		 
      [2021-03-04 16:00:11,554] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 13.00%
      		 
      [2021-03-04 16:00:22,243] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:00:25,066] - [rest_client] [139888393434880] - INFO - rebalance percentage : 14.00 %
      		 
      [2021-03-04 16:00:25,067] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 14.00%
      		 
      [2021-03-04 16:00:35,704] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:00:37,816] - [rest_client] [139888393434880] - INFO - rebalance percentage : 15.00 %
      		 
      [2021-03-04 16:00:37,816] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 15.00%
      		 
      [2021-03-04 16:00:49,338] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:00:51,979] - [rest_client] [139888393434880] - INFO - rebalance percentage : 16.00 %
      		 
      [2021-03-04 16:00:51,979] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 16.00%
      		 
      [2021-03-04 16:01:03,528] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:01:05,850] - [rest_client] [139888393434880] - INFO - rebalance percentage : 16.00 %
      		 
      [2021-03-04 16:01:05,852] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 16.00%
      		 
      [2021-03-04 16:01:17,427] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:01:19,433] - [rest_client] [139888393434880] - INFO - rebalance percentage : 18.00 %
      		 
      [2021-03-04 16:01:19,434] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 18.00%
      		 
      [2021-03-04 16:01:29,836] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:01:31,146] - [rest_client] [139888393434880] - INFO - rebalance percentage : 19.00 %
      		 
      [2021-03-04 16:01:31,147] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 19.00%
      		 
      [2021-03-04 16:01:41,586] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:01:44,106] - [rest_client] [139888393434880] - INFO - rebalance percentage : 19.00 %
      		 
      [2021-03-04 16:01:44,106] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 19.00%
      		 
      [2021-03-04 16:01:55,128] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:01:57,780] - [rest_client] [139888393434880] - INFO - rebalance percentage : 21.00 %
      		 
      [2021-03-04 16:01:57,780] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 21.00%
      		 
      [2021-03-04 16:02:08,633] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:02:08,972] - [rest_client] [139888393434880] - INFO - rebalance percentage : 22.00 %
      		 
      [2021-03-04 16:02:08,973] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 22.00%
      		 
      [2021-03-04 16:02:18,999] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:02:20,812] - [rest_client] [139888393434880] - INFO - rebalance percentage : 22.00 %
      		 
      [2021-03-04 16:02:20,812] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 22.00%
      		 
      [2021-03-04 16:02:32,241] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:02:34,607] - [rest_client] [139888393434880] - INFO - rebalance percentage : 23.00 %
      		 
      [2021-03-04 16:02:34,607] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 23.00%
      		 
      [2021-03-04 16:02:46,048] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:02:48,308] - [rest_client] [139888393434880] - INFO - rebalance percentage : 24.00 %
      		 
      [2021-03-04 16:02:48,308] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 24.00%
      		 
      [2021-03-04 16:02:59,336] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:03:01,654] - [rest_client] [139888393434880] - INFO - rebalance percentage : 25.00 %
      		 
      [2021-03-04 16:03:01,655] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 25.00%
      		 
      [2021-03-04 16:03:13,455] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:03:16,893] - [rest_client] [139888393434880] - INFO - rebalance percentage : 27.00 %
      		 
      [2021-03-04 16:03:16,893] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 27.00%
      		 
      [2021-03-04 16:03:27,130] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:03:29,273] - [rest_client] [139888393434880] - INFO - rebalance percentage : 27.00 %
      		 
      [2021-03-04 16:03:29,274] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 27.00%
      		 
      [2021-03-04 16:03:40,105] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:03:43,214] - [rest_client] [139888393434880] - INFO - rebalance percentage : 29.00 %
      		 
      [2021-03-04 16:03:43,214] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 29.00%
      		 
      [2021-03-04 16:03:54,874] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:03:56,317] - [rest_client] [139888393434880] - INFO - rebalance percentage : 30.00 %
      		 
      [2021-03-04 16:03:56,317] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 30.00%
      		 
      [2021-03-04 16:04:07,454] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:04:09,996] - [rest_client] [139888393434880] - INFO - rebalance percentage : 31.00 %
      		 
      [2021-03-04 16:04:09,996] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 31.00%
      		 
      [2021-03-04 16:04:20,894] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:04:24,051] - [rest_client] [139888393434880] - INFO - rebalance percentage : 32.00 %
      		 
      [2021-03-04 16:04:24,051] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 32.00%
      		 
      [2021-03-04 16:04:34,989] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:04:35,317] - [rest_client] [139888393434880] - INFO - rebalance percentage : 33.00 %
      		 
      [2021-03-04 16:04:35,317] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 33.00%
      		 
      [2021-03-04 16:04:45,342] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:04:45,688] - [rest_client] [139888393434880] - INFO - rebalance percentage : 38.00 %
      		 
      [2021-03-04 16:04:45,689] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 38.00%
      		 
      [2021-03-04 16:04:55,714] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:04:55,871] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:04:55,871] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:05:05,894] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:05:06,205] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:05:06,206] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:05:16,227] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:05:16,570] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:05:16,570] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:05:26,592] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:05:26,741] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:05:26,742] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:05:36,762] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:05:37,086] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:05:37,087] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:05:47,110] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:05:47,422] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:05:47,422] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:05:57,445] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:05:57,614] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:05:57,614] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:06:07,635] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:06:07,945] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:06:07,945] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:06:17,967] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:06:18,283] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:06:18,283] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:06:28,305] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:06:28,631] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:06:28,632] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:06:38,654] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:06:38,810] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:06:38,810] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:06:48,834] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:06:49,149] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:06:49,149] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:06:59,177] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:06:59,502] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:06:59,502] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:07:09,525] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:07:09,687] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:07:09,687] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:07:19,710] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:07:20,028] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:07:20,028] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:07:30,056] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:07:30,373] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:07:30,373] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:07:40,397] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:07:40,558] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:07:40,558] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:07:50,580] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:07:50,892] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:07:50,894] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:08:00,922] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:08:01,234] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:08:01,234] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:08:11,258] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:08:11,454] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:08:11,455] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:08:21,479] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:08:21,780] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:08:21,780] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:08:31,802] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:08:32,112] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:08:32,112] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:08:42,135] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:08:42,457] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:08:42,458] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:08:52,480] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:08:52,650] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:08:52,650] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:09:02,676] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:09:02,988] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:09:02,988] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:09:13,013] - [task] [139888393434880] - INFO - This is swap rebalance and we will monitor vbuckets shuffling
      		 
      [2021-03-04 16:09:13,332] - [rest_client] [139888393434880] - INFO - rebalance percentage : 42.00 %
      		 
      [2021-03-04 16:09:13,333] - [task] [139888393434880] - INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2021-03-04 16:09:13,349] - [rest_client] [139888393434880] - INFO - Latest logs from UI on 172.23.100.11:
      		 
      [2021-03-04 16:09:13,350] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.14', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1614902675992, 'shortText': 'message', 'text': 'Shutting down bucket "default" on \'ns_1@172.23.100.14\' for deletion', 'serverTime': '2021-03-04T16:04:35.992Z'}
      		 
      [2021-03-04 16:09:13,350] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'ns_vbucket_mover', 'tstamp': 1614902533680, 'shortText': 'message', 'text': 'Bucket "default" rebalance appears to be swap rebalance', 'serverTime': '2021-03-04T16:02:13.680Z'}
      		 
      [2021-03-04 16:09:13,350] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.14', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1614902532408, 'shortText': 'message', 'text': 'Shutting down bucket "standard_bucket0" on \'ns_1@172.23.100.14\' for deletion', 'serverTime': '2021-03-04T16:02:12.408Z'}
      		 
      [2021-03-04 16:09:13,350] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.15', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1614902532325, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.100.15\' in 0 seconds.', 'serverTime': '2021-03-04T16:02:12.325Z'}
      		 
      [2021-03-04 16:09:13,351] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'ns_rebalancer', 'tstamp': 1614902532255, 'shortText': 'message', 'text': 'Started rebalancing bucket default', 'serverTime': '2021-03-04T16:02:12.255Z'}
      		 
      [2021-03-04 16:09:13,351] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'ns_vbucket_mover', 'tstamp': 1614902380032, 'shortText': 'message', 'text': 'Bucket "standard_bucket0" rebalance appears to be swap rebalance', 'serverTime': '2021-03-04T15:59:40.032Z'}
      		 
      [2021-03-04 16:09:13,351] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.14', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1614902378654, 'shortText': 'message', 'text': 'Shutting down bucket "travel-sample" on \'ns_1@172.23.100.14\' for deletion', 'serverTime': '2021-03-04T15:59:38.654Z'}
      		 
      [2021-03-04 16:09:13,351] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.15', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1614902378607, 'shortText': 'message', 'text': 'Bucket "standard_bucket0" loaded on node \'ns_1@172.23.100.15\' in 0 seconds.', 'serverTime': '2021-03-04T15:59:38.607Z'}
      		 
      [2021-03-04 16:09:13,352] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'ns_rebalancer', 'tstamp': 1614902378525, 'shortText': 'message', 'text': 'Started rebalancing bucket standard_bucket0', 'serverTime': '2021-03-04T15:59:38.525Z'}
      		 
      [2021-03-04 16:09:13,352] - [rest_client] [139888393434880] - ERROR - {'node': 'ns_1@172.23.100.11', 'type': 'info', 'code': 0, 'module': 'ns_vbucket_mover', 'tstamp': 1614902223819, 'shortText': 'message', 'text': 'Bucket "travel-sample" rebalance appears to be swap rebalance', 'serverTime': '2021-03-04T15:57:03.819Z'}

      Attached server logs for each test occurrence.

       

      Attachments

        1. test_11.zip
          41.03 MB
        2. test_3.zip
          50.40 MB
        3. test_6.zip
          40.73 MB
        4. test_7.zip
          40.74 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            kevin.cherkauer Kevin Cherkauer (Inactive)
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty