Critical Description
Build
4.5.0-1673
Testcase
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-logs=False,stop-on-failure=False,cluster=D+F:F:F,GROUP=P0 -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=1000,cluster=D,F,F,standard_buckets=3,sasl_buckets=3,index_per_bucket=3,GROUP=P0
Rebalance fails during teardown phase of a test where 21 indexes are first deleted, followed by 7 buckets and 2 fts nodes[.145, .234 ] are kicked out from the cluster [.187]
[2016-03-04 10:55:48,001] - [bucket_helper:137] INFO - deleting existing buckets [u'default', u'sasl_bucket_1', u'sasl_bucket_2', u'sasl_bucket_3', u'standard_bucket_1', u'standard_bucket_2', u'standard_bucket_3'] on 172.23.107.187
|
[2016-03-04 10:55:48,001] - [bucket_helper:139] INFO - remove bucket default ...
|
[2016-03-04 10:55:50,000] - [bucket_helper:153] INFO - deleted bucket : default from 172.23.107.187
|
[2016-03-04 10:55:50,001] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:55:50,113] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : [u'sasl_bucket_1', u'sasl_bucket_2', u'sasl_bucket_3', u'standard_bucket_1', u'standard_bucket_2', u'standard_bucket_3']
|
[2016-03-04 10:55:50,115] - [bucket_helper:139] INFO - remove bucket sasl_bucket_1 ...
|
[2016-03-04 10:55:54,708] - [bucket_helper:153] INFO - deleted bucket : sasl_bucket_1 from 172.23.107.187
|
[2016-03-04 10:55:54,709] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:55:54,819] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : [u'sasl_bucket_2', u'sasl_bucket_3', u'standard_bucket_1', u'standard_bucket_2', u'standard_bucket_3']
|
[2016-03-04 10:55:54,820] - [bucket_helper:139] INFO - remove bucket sasl_bucket_2 ...
|
[2016-03-04 10:56:00,466] - [bucket_helper:153] INFO - deleted bucket : sasl_bucket_2 from 172.23.107.187
|
[2016-03-04 10:56:00,466] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:56:04,213] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : [u'sasl_bucket_3', u'standard_bucket_1', u'standard_bucket_2', u'standard_bucket_3']
|
[2016-03-04 10:56:04,214] - [bucket_helper:139] INFO - remove bucket sasl_bucket_3 ...
|
[2016-03-04 10:56:05,674] - [bucket_helper:153] INFO - deleted bucket : sasl_bucket_3 from 172.23.107.187
|
[2016-03-04 10:56:05,674] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:56:05,799] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : [u'standard_bucket_1', u'standard_bucket_2', u'standard_bucket_3']
|
[2016-03-04 10:56:05,800] - [bucket_helper:139] INFO - remove bucket standard_bucket_1 ...
|
[2016-03-04 10:56:07,824] - [bucket_helper:153] INFO - deleted bucket : standard_bucket_1 from 172.23.107.187
|
[2016-03-04 10:56:07,824] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:56:07,859] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : [u'standard_bucket_2', u'standard_bucket_3']
|
[2016-03-04 10:56:07,860] - [bucket_helper:139] INFO - remove bucket standard_bucket_2 ...
|
[2016-03-04 10:56:15,253] - [bucket_helper:153] INFO - deleted bucket : standard_bucket_2 from 172.23.107.187
|
[2016-03-04 10:56:15,253] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:56:15,297] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : [u'standard_bucket_3']
|
[2016-03-04 10:56:15,298] - [bucket_helper:139] INFO - remove bucket standard_bucket_3 ...
|
[2016-03-04 10:56:16,261] - [bucket_helper:153] INFO - deleted bucket : standard_bucket_3 from 172.23.107.187
|
[2016-03-04 10:56:16,262] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
|
[2016-03-04 10:56:16,266] - [rest_client:127] INFO - node 172.23.107.187 existing buckets : []
|
[2016-03-04 10:56:23,903] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
|
[2016-03-04 10:56:23,910] - [rest_client:1206] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.107.145%2Cns_1%40172.23.107.234&user=Administrator&knownNodes=ns_1%40172.23.107.187%2Cns_1%40172.23.107.145%2Cns_1%40172.23.107.234
|
[2016-03-04 10:56:23,919] - [rest_client:1210] INFO - rebalance operation started
|
[2016-03-04 10:56:27,651] - [rest_client:1338] INFO - rebalance percentage : 0.00 %
|
[2016-03-04 10:56:36,461] - [rest_client:1321] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
|
Attaching cbcollect from all nodes.