Description
Build
5.0.0-3158
This test last passed on build 5.0.0-3136
Please note this build contains level compaction code(turned off by default)
Testcase
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P0 -t fts.moving_topology_fts.MovingTopFTS.rebalance_in_during_index_building,items=100000,cluster=D,F,GROUP=P0
A basic rebalance-in of a single node(.54) into [D,F] cluster during indexing of 100K docs. Please note all subsequent rebalances on this cluster fail.
Index building has begun...
|
2017-06-21 11:31:40 | INFO | MainProcess | test_thread | [moving_topology_fts.rebalance_in_during_index_building] Index count for default_index_1: 5331 |
2017-06-21 11:31:40 | INFO | MainProcess | test_thread | [fts_base.async_rebalance_in] Starting rebalance-in nodes:[ip:172.23.105.54 port:8091 ssh_username:root] at C1 cluster 172.23.105.96 |
2017-06-21 11:31:40 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.54:8091 to cluster |
2017-06-21 11:31:40 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.54:8091 to this cluster @172.23.105.96:8091 |
2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.54,ns_1@172.23.105.96,ns_1@172.23.105.190'} |
2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started |
2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % |
2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0 |
2017-06-21 11:31:56 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 10.50 % |
2017-06-21 11:31:56 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 10.5034722222 |
2017-06-21 11:32:07 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 22.94 % |
2017-06-21 11:32:07 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 22.9383680556 |
2017-06-21 11:32:17 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 36.85 % |
2017-06-21 11:32:17 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 36.8489583333 |
2017-06-21 11:32:27 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % |
2017-06-21 11:32:27 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 50.0 |
2017-06-21 11:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % |
2017-06-21 11:32:37 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 50.0 |
2017-06-21 11:32:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed |
2017-06-21 11:32:47 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.105.96: |
2017-06-21 11:32:47 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.96', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n {badmatch,\n {error,\n {bad_nodes,fts,get_node_infos,\n [{\'ns_1@172.23.105.190\',\n {exit,\n {{linked_process_died,<20245.8315.0>,\n {no_connection,"fts-service_api"}},\n {gen_server,call,\n [{\'service_agent-fts\',\n \'ns_1@172.23.105.190\'},\n {if_rebalance,<0.11100.2>,\n get_node_info},\n infinity]}}}}]}}}}', u'shortText': u'message', u'serverTime': u'2017-06-21T11:32:41.178Z', u'module': u'ns_orchestrator', u'tstamp': 1498069961178, u'type': u'critical'} |
Attachments
Issue Links
- duplicates
-
MB-24976 [FTS] moss infinite loop in determineExponent()?
- Closed