Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.0.0
-
Security Level: Public
-
couchbase-server-enterprise-4.0.0-4043
-
Untriaged
-
Centos 64-bit
-
Yes
Description
1. Create a cluster of 6 nodes
2. Create a default bucket
3. Add 100000 items
4. Make one of the nodes unavailable by changing access to ports by changing iptables
2015-08-14 03:19:23 | INFO | MainProcess | test_thread | [rest_client.wait_for_node_status] node ns_1@172.23.106.108 status : unhealthy
2015-08-14 03:19:23 | INFO | MainProcess | test_thread | [rest_client.wait_for_node_status] node ns_1@172.23.106.108 status_reached : True
2015-08-14 03:19:23 | INFO | MainProcess | test_thread | [failovertests.run_failover_operations] node 172.23.106.108:8091 is 'unhealthy' as expected
2015-08-14 03:20:25 | INFO | MainProcess | test_thread | [rest_client.fail_over] fail_over node ns_1@172.23.106.108 successful
5. Now rebalance and eject the node from the cluster:
2015-08-14 03:20:35 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.108&user=Administrator&knownNodes=ns_1%40172.23.106.108%2Cns_1%40172.23.106.111%2Cns_1%40172.23.106.109%2Cns_1%40172.23.106.107%2Cns_1%40172.23.106.112%2Cns_1%40172.23.106.110
Rebalance fails:
2015-08-14 03:20:35 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
2015-08-14 03:20:35 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:20:40 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:20:45 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:20:50 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:20:55 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:00 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:05 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:10 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:15 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:20 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:25 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:30 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client._rebalance_progress]
- rebalance failed
2015-08-14 03:21:35 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.106.107:
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.109', u'code': 5, u'text': u"Node 'ns_1@172.23.106.109' saw that node 'ns_1@172.23.106.108' went down. Details: [
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.112', u'code': 5, u'text': u"Node 'ns_1@172.23.106.112' saw that node 'ns_1@172.23.106.108' went down. Details: [{nodedown_reason,n net_tick_timeout}
]", u'shortText': u'node down', u'serverTime': u'2015-08-14T03:27:37.221Z', u'module': u'ns_node_disco', u'tstamp': 1439548057221, u'type': u'warning'}
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.111', u'code': 5, u'text': u"Node 'ns_1@172.23.106.111' saw that node 'ns_1@172.23.106.108' went down. Details: [
]", u'shortText': u'node down', u'serverTime': u'2015-08-14T03:27:33.716Z', u'module': u'ns_node_disco', u'tstamp': 1439548053716, u'type': u'warning'}
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.106.109', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.109\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-08-14T03:22:51.396Z', u'module': u'ns_memcached', u'tstamp': 1439547771396, u'type': u'info'}2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.106.111', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.111\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-08-14T03:22:50.939Z', u'module': u'ns_memcached', u'tstamp': 1439547770939, u'type': u'info'}2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.106.112', u'code': 3, u'text': u'Node ns_1@172.23.106.112 joined cluster', u'shortText': u'message', u'serverTime': u'2015-08-14T03:22:40.661Z', u'module': u'ns_cluster', u'tstamp': 1439547760661, u'type': u'info'}2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.106.112', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.106.112\'. Version: "4.0.0-4043-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2015-08-14T03:22:40.612Z', u'module': u'menelaus_sup', u'tstamp': 1439547760612, u'type': u'info'}2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.106.111', u'code': 4, u'text': u"Node 'ns_1@172.23.106.111' saw that node 'ns_1@172.23.106.112' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2015-08-14T03:22:37.756Z', u'module': u'ns_node_disco', u'tstamp': 1439547757756, u'type': u'info'}2015-08-14 03:21:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.106.109', u'code': 4, u'text': u"Node 'ns_1@172.23.106.109' saw that node 'ns_1@172.23.106.112' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2015-08-14T03:22:37.710Z', u'module': u'ns_node_disco', u'tstamp': 1439547757710, u'type': u'info'}ERROR
Regression from build 4019.
Attachments
Issue Links
- blocks
-
MB-15886 3.1.1 Minor Release
- Resolved
For Gerrit Dashboard: MB-16064 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
54514,2 | Revert "MB-15926 Properly close last replication stream(s)." | rel-3.0.0++ | ns_server | Status: MERGED | +2 | +1 |
54515,2 | MB-16064 Shut down only consumers when terminating replictor. | rel-3.0.0++ | ns_server | Status: MERGED | +2 | +1 |
54529,1 | Merge remote-tracking branch 'couchbase/rel-3.0.0++' into sherlock | sherlock | ns_server | Status: MERGED | +2 | +1 |
54532,3 | MB-16064: Advance ns_server SHA in sherlock-4.0.0 manifest | sherlock-4.0.0 | manifest | Status: MERGED | +2 | +1 |
54552,1 | Merge remote-tracking branch 'couchbase/sherlock' | master | ns_server | Status: MERGED | +2 | +1 |