Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 3.1.1, 4.0.0
Affects Version/s: 4.0.0
Component/s: ns_server
Security Level: Public
Labels:
- ga
Environment:
couchbase-server-enterprise-4.0.0-4043

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://s3.amazonaws.com/bugdb/jira/MB-16064/logs.tar
Is this a Regression?:
Yes

Description

1. Create a cluster of 6 nodes
2. Create a default bucket
3. Add 100000 items
4. Make one of the nodes unavailable by changing access to ports by changing iptables
2015-08-14 03:19:23 | INFO | MainProcess | test_thread | [rest_client.wait_for_node_status] node ns_1@172.23.106.108 status : unhealthy
2015-08-14 03:19:23 | INFO | MainProcess | test_thread | [rest_client.wait_for_node_status] node ns_1@172.23.106.108 status_reached : True
2015-08-14 03:19:23 | INFO | MainProcess | test_thread | [failovertests.run_failover_operations] node 172.23.106.108:8091 is 'unhealthy' as expected
2015-08-14 03:20:25 | INFO | MainProcess | test_thread | [rest_client.fail_over] fail_over node ns_1@172.23.106.108 successful
5. Now rebalance and eject the node from the cluster:
2015-08-14 03:20:35 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.108&user=Administrator&knownNodes=ns_1%40172.23.106.108%2Cns_1%40172.23.106.111%2Cns_1%40172.23.106.109%2Cns_1%40172.23.106.107%2Cns_1%40172.23.106.112%2Cns_1%40172.23.106.110

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
2015-08-14 03:21:35 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.106.107:
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.109', u'code': 5, u'text': u"Node 'ns_1@172.23.106.109' saw that node 'ns_1@172.23.106.108' went down. Details: [

{nodedown_reason,\n net_tick_timeout}]", u'shortText': u'node down', u'serverTime': u'2015-08-14T03:27:40.750Z', u'module': u'ns_node_disco', u'tstamp': 1439548060750, u'type': u'warning'}
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.112', u'code': 5, u'text': u"Node 'ns_1@172.23.106.112' saw that node 'ns_1@172.23.106.108' went down. Details: [{nodedown_reason,n net_tick_timeout}

]", u'shortText': u'node down', u'serverTime': u'2015-08-14T03:27:37.221Z', u'module': u'ns_node_disco', u'tstamp': 1439548057221, u'type': u'warning'}
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.111', u'code': 5, u'text': u"Node 'ns_1@172.23.106.111' saw that node 'ns_1@172.23.106.108' went down. Details: [

{nodedown_reason,\n net_tick_timeout}

]", u'shortText': u'node down', u'serverTime': u'2015-08-14T03:27:33.716Z', u'module': u'ns_node_disco', u'tstamp': 1439548053716, u'type': u'warning'}
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]

{u'node': u'ns_1@172.23.106.112', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.112\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-08-14T03:22:51.452Z', u'module': u'ns_memcached', u'tstamp': 1439547771452, u'type': u'info'}