Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 4.5.0
Affects Version/s: 4.5.0
Component/s: cbft
Labels:
None

Triage:
Untriaged
Is this a Regression?:
Unknown

Description

Build
4.5.0-2011

Testcase
test_29 in windows fts rebalance suite

./testrunner -i /tmp/testexec.27410.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,items=10000,query_max_matches=10000 -t fts.moving_topology_fts.MovingTopFTS.rebalance_in_between_indexing_and_querying,items=30000,cluster=D,F,D+F,sasl_buckets=1,GROUP=P0

Rebalance fails while trying to add 172.23.107.37 to cluster 172.23.106.181.

[2016-04-04 17:26:23,779] - [fts_base:1621] INFO - Starting rebalance-in nodes:[ip:172.23.107.37 port:8091 ssh_username:Administrator] at C1 cluster 172.23.106.181

[2016-04-04 17:26:24,535] - [task:404] INFO - adding node 172.23.107.37:8091 to cluster

[2016-04-04 17:26:24,536] - [rest_client:988] INFO - adding remote node @172.23.107.37:8091 to this cluster @172.23.106.181:8091

[2016-04-04 17:26:29,008] - [rest_client:1206] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.146%2Cns_1%40172.23.106.182%2Cns_1%40172.23.106.181%2Cns_1%40172.23.107.37

[2016-04-04 17:26:29,013] - [rest_client:1210] INFO - rebalance operation started

[2016-04-04 17:26:29,018] - [rest_client:1338] INFO - rebalance percentage : 0.00 %

[2016-04-04 17:26:39,038] - [rest_client:1338] INFO - rebalance percentage : 10.82 %

[2016-04-04 17:26:49,057] - [rest_client:1338] INFO - rebalance percentage : 23.86 %

[2016-04-04 17:26:59,127] - [rest_client:1338] INFO - rebalance percentage : 34.38 %

[2016-04-04 17:27:09,145] - [rest_client:1338] INFO - rebalance percentage : 46.67 %

[2016-04-04 17:27:19,164] - [rest_client:1321] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed

[2016-04-04 17:27:19,249] - [rest_client:2443] INFO - Latest logs from UI on 172.23.106.181:

[2016-04-04 17:27:19,249] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 2, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n                              {rebalance_failed,\n                               {service_error,\n                                <<"invalid character \'N\' looking for beginning of value">>}}}\n', u'shortText': u'message', u'serverTime': u'2016-04-04T17:27:12.926Z', u'module': u'ns_orchestrator', u'tstamp': 1459816032926, u'type': u'info'}

[2016-04-04 17:27:19,249] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:50.928Z', u'module': u'ns_vbucket_mover', u'tstamp': 1459816010928, u'type': u'info'}

[2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.37\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:50.091Z', u'module': u'ns_memcached', u'tstamp': 1459816010091, u'type': u'info'}

[2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:49.769Z', u'module': u'ns_rebalancer', u'tstamp': 1459816009769, u'type': u'info'}

[2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Bucket "sasl_bucket_1" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:30.158Z', u'module': u'ns_vbucket_mover', u'tstamp': 1459815990158, u'type': u'info'}

[2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 0, u'text': u'Bucket "sasl_bucket_1" loaded on node \'ns_1@172.23.107.37\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:29.859Z', u'module': u'ns_memcached', u'tstamp': 1459815989859, u'type': u'info'}

[2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 3, u'text': u'Node ns_1@172.23.107.37 joined cluster', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:29.207Z', u'module': u'ns_cluster', u'tstamp': 1459815989207, u'type': u'info'}

[2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.37\'. Version: "4.5.0-2011-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2016-04-04T17:26:29.147Z', u'module': u'menelaus_sup', u'tstamp': 1459815989147, u'type': u'info'}

[2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Started rebalancing bucket sasl_bucket_1', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:28.997Z', u'module': u'ns_rebalancer', u'tstamp': 1459815988997, u'type': u'info'}

[2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.146','ns_1@172.23.106.182',\n                                 'ns_1@172.23.106.181','ns_1@172.23.107.37'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:28.955Z', u'module': u'ns_orchestrator', u'tstamp': 1459815988955, u'type': u'info'}