Details
-
Bug
-
Resolution: Duplicate
-
Test Blocker
-
5.5.0
-
None
-
5.5.0-1659
-
Untriaged
-
Centos 64-bit
-
-
Yes
Description
- Setup a node with Vulcan and bucket with 1000 items
- Rebalance in 3 nodes.
- Rebalance is very slow starting from build
5.5.0-16625.5.0-1659 - Last good build was 5.0.0-1654.
Build - 5.5.0-1662
2018-01-10 04:07:03 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.109.134,ns_1@172.23.109.132,ns_1@172.23.109.133,ns_1@172.23.109.131'}
2018-01-10 04:07:03 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2018-01-10 04:07:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2018-01-10 04:07:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0
2018-01-10 04:07:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2018-01-10 04:07:13 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0
2018-01-10 04:07:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.21 %
2018-01-10 04:07:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.208333333333
2018-01-10 04:07:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.21 %
2018-01-10 04:07:33 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.208333333333
2018-01-10 04:07:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.31 %
2018-01-10 04:07:43 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.3125
2018-01-10 04:07:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.52 %
2018-01-10 04:07:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.520833333333
2018-01-10 04:08:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.62 %
2018-01-10 04:08:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.625
2018-01-10 04:08:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.83 %
2018-01-10 04:08:13 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.833333333333
2018-01-10 04:08:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.04 %
2018-01-10 04:08:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.04166666667
2018-01-10 04:08:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.25 %
2018-01-10 04:08:33 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.25
2018-01-10 04:08:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.46 %
2018-01-10 04:08:43 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.45833333333
2018-01-10 04:08:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.46 %
2018-01-10 04:08:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.45833333333
2018-01-10 04:09:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.67 %
2018-01-10 04:09:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.66666666667
2018-01-10 04:09:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.77 %
2018-01-10 04:09:13 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.77083333333
2018-01-10 04:09:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 1.98 %
2018-01-10 04:09:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 1.97916666667
2018-01-10 04:09:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 2.08 %
2018-01-10 04:09:33 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 2.08333333333
2018-01-10 04:09:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 2.29 %
2018-01-10 04:09:43 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 2.29166666667
2018-01-10 04:09:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 2.40 %
2018-01-10 04:09:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 2.39583333333
2018-01-10 04:10:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 2.60 %
2018-01-10 04:10:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 2.60416666667
2018-01-10 04:10:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 2.81 %
2018-01-10 04:10:13 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 2.8125
2018-01-10 04:10:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 3.02 %
2018-01-10 04:10:24 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 3.02083333333
2018-01-10 04:10:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 3.13 %
2018-01-10 04:10:34 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 3.125
2018-01-10 04:10:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 3.23 %
2018-01-10 04:10:44 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 3.22916666667
2018-01-10 04:10:54 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 3.44 %
2018-01-10 04:10:54 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 3.4375
2018-01-10 04:11:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 3.54 %
2018-01-10 04:11:04 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 3.54166666667
2018-01-10 04:11:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 3.75 %
2018-01-10 04:11:14 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 3.75
2018-01-10 04:11:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.06 %
2018-01-10 04:11:24 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.0625
2018-01-10 04:11:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.06 %
2018-01-10 04:11:34 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.0625
2018-01-10 04:11:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.27 %
2018-01-10 04:11:44 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.27083333333
2018-01-10 04:11:54 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.27 %
2018-01-10 04:11:54 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.27083333333
2018-01-10 04:12:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.48 %
2018-01-10 04:12:04 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.47916666667
2018-01-10 04:12:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.69 %
2018-01-10 04:12:14 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.6875
2018-01-10 04:12:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 4.79 %
2018-01-10 04:12:24 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 4.79166666667
Same test case with build 5.5.0-1651
2018-01-08 23:33:15 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.109.134:8091 to this cluster @172.23.109.131:8091
2018-01-08 23:33:20 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.109.134,ns_1@172.23.109.132,ns_1@172.23.109.133,ns_1@172.23.109.131'}
2018-01-08 23:33:20 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2018-01-08 23:33:20 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %
2018-01-08 23:33:20 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0
2018-01-08 23:33:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 23.75 %
2018-01-08 23:33:30 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 23.75
2018-01-08 23:33:40 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 52.29 %
2018-01-08 23:33:40 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 52.2916666667
2018-01-08 23:33:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 80.26 %
2018-01-08 23:33:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 80.2604166667
2018-01-08 23:34:00 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100
2018-01-08 23:34:00 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 40.1905429363 sec