Critical Description
Build
4.5.0-1590
Testcase (not merged yet)
./testrunner -i INI_FILE.ini -p skip-cleanup=True,get-cbcollect-info=False,get-logs=False,stop-on-failure=False -t fts.moving_topology_fts.MovingTopFTS.rebalance_out_during_index_building,items=100000,cluster=D,F,F,GROUP=P0
Steps
1. To a single data node cluster(.136), add 2 fts nodes [.175, .176]
2. Create a default index, load 100K docs
3. Index building begins now. After 10s, rebalance out one of the fts nodes[.176].
I can see that while index building is progressing without any problems, rebalance is stuck. There is no sign of pindexes being moved over while indexing continues. Screenshot shows 0% progress on .176 and .175.
Test log:
2016-02-26 09:29:38 | INFO | MainProcess | test_thread | [fts_base.create] Creating fulltext-index default_index_1 on 172.23.106.175
|
2016-02-26 09:29:38 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] {"params": {}, "name": "default_index_1", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""}
|
2016-02-26 09:29:38 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] Index default_index_1 created
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Number of PIndexes = 32
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Every pIndex serves 32 partitions or lesser
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: pIndexes are distributed across [u'06f0f79338abd60607770d2b72b0ee73', u'ff9b1cbd0275758439e65d4ff05a41c2']
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Expecting num of partitions in each node in range 480-544
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node 06f0f79338abd60607770d2b72b0ee73 houses 16 pindexes which serve 512 partitions
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node ff9b1cbd0275758439e65d4ff05a41c2 houses 16 pindexes which serve 512 partitions
|
2016-02-26 09:29:39 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 10 secs. ...
|
2016-02-26 09:29:49 | INFO | MainProcess | test_thread | [moving_topology_fts.rebalance_out_during_index_building] Index building has begun...
|
2016-02-26 09:29:49 | INFO | MainProcess | test_thread | [fts_base.__async_rebalance_out] Starting rebalance-out nodes:[ip:172.23.106.176 port:8091 ssh_username:root] at C1 cluster 172.23.106.139
|
2016-02-26 09:29:50 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.176&user=Administrator&knownNodes=ns_1%40172.23.106.176%2Cns_1%40172.23.106.175%2Cns_1%40172.23.106.139
|
2016-02-26 09:29:50 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
|
2016-02-26 09:29:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
|
2016-02-26 09:30:00 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:30:10 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:30:20 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:30:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:30:51 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:31:01 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:31:11 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:31:21 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:31:31 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:31:41 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:31:51 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:32:01 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:32:11 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:32:21 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:32:31 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:32:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:32:52 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:33:02 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:33:12 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:33:22 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:33:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:33:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:33:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:34:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:34:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:34:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:34:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:34:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:34:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:35:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:35:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:35:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:35:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:35:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:36:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:36:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:36:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:36:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:36:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:36:54 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:37:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:37:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:37:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:37:34 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:37:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:37:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:38:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:38:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:38:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:38:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:38:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:38:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:39:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:39:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:39:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:39:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:39:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:39:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:40:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
2016-02-26 09:40:16 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 %
|
Attaching screenshot and cbcollect info.
