Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
4.5.1
-
Untriaged
-
Unknown
Description
Build - 4.5.1-2838
Seeing an issue with bucket deletion during cleanup in the below mentioned test. This is similar to MB-20751
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P1 -t fts.stable_topology_fts.StableTopFTS.index_wiki,items=20000,lang=ES,analyzer=es,cluster=D+F,F,F,GROUP=P1
Job : http://qa.sc.couchbase.com/job/cen006-p1-fts-vset00-02-stable-topology/358/consoleFull (test_10)
Test log snippet :
[2016-09-05 21:15:25,686] - [fts_base:806] INFO - Validated: all index files for wiki_index deleted from disk
|
[2016-09-05 21:15:25,686] - [fts_base:1117] INFO - removing nodes from cluster ...
|
[2016-09-05 21:15:25,696] - [fts_base:1119] INFO - cleanup [ip:172.23.107.187 port:8091 ssh_username:root, ip:172.23.107.145 port:8091 ssh_username:root, ip:172.23.107.234 port:8091 ssh_username:root]
|
[2016-09-05 21:15:25,777] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.107.187
|
[2016-09-05 21:15:25,777] - [bucket_helper:140] INFO - remove bucket default ...
|
[2016-09-05 21:15:55,793] - [rest_client:778] ERROR - http://172.23.107.187:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
|
[2016-09-05 21:15:55,794] - [rest_client:1852] WARNING - Bucket deletion timed out waiting for all nodes
|
[2016-09-05 21:15:55,895] - [bucket_helper:153] ERROR - Unable to get timings for bucket
|
[2016-09-05 21:15:55,895] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.107.187
|
[2016-09-05 21:15:55,895] - [bucket_helper:230] INFO - waiting for bucket deletion to complete....
|
[2016-09-05 21:15:55,901] - [rest_client:133] INFO - node 172.23.107.187 existing buckets : []
|
[2016-09-05 21:15:55,951] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
|
[2016-09-05 21:15:55,954] - [rest_client:1292] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.107.145%2Cns_1%40172.23.107.234&user=Administrator&knownNodes=ns_1%40172.23.107.187%2Cns_1%40172.23.107.145%2Cns_1%40172.23.107.234
|
[2016-09-05 21:15:55,960] - [rest_client:1296] INFO - rebalance operation started
|
[2016-09-05 21:15:55,979] - [rest_client:1427] INFO - rebalance percentage : 0.00 %
|
[2016-09-05 21:16:05,993] - [rest_client:1427] INFO - rebalance percentage : 0.00 %
|
[2016-09-05 21:16:16,007] - [rest_client:1410] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
|
VMs:
172.23.107.234
172.23.107.187
172.23.107.145
Attaching server logs shortly