Details
Description
http://qa.hq.northscale.net/job/windows_rebalance-kv_P0/72/consoleFull
Alk, not sure that this is a bug, maybe we need to retry here?
when we rebalance out the cluster we check that 'pools' is empty on nodes via rest api,
but couchApiBase doesn't exist in nodes/self:
013-06-06 08:13:43 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2013-06-06 08:13:43 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%4010.1.3.83%2Cns_1%4010.1.3.85%2Cns_1%4010.1.3.84%2Cns_1%4010.1.3.86%2Cns_1%4010.1.3.87&user=Administrator&knownNodes=ns_1%4010.1.3.83%2Cns_1%4010.1.3.85%2Cns_1%4010.1.3.84%2Cns_1%4010.1.3.82%2Cns_1%4010.1.3.86%2Cns_1%4010.1.3.87
2013-06-06 08:13:43 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started
2013-06-06 08:13:43 | INFO | MainProcess | MainThread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-06-06 08:13:48 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 4.03037905693 seconds
2013-06-06 08:13:48 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 4.03037905693 seconds after rebalance...
2013-06-06 08:13:52 | ERROR | MainProcess | MainThread | [rest_client._http_request] http://10.1.3.87:8091/pools/default error 404 reason: unknown "unknown pool"
http://10.1.3.87:8091/pools/default with status False: unknown pool
2013-06-06 08:13:52 | ERROR | MainProcess | MainThread | [rest_client._http_request] http://10.1.3.87:8091/pools/default error 404 reason: unknown "unknown pool"
http://10.1.3.87:8091/pools/default with status False: unknown pool
2013-06-06 08:13:52 | ERROR | MainProcess | MainThread | [rest_client._http_request] http://10.1.3.87:8091/pools/default error 404 reason: unknown "unknown pool"
http://10.1.3.87:8091/pools/default with status False: unknown pool
2013-06-06 08:13:52 | ERROR | MainProcess | MainThread | [rest_client._http_request] http://10.1.3.87:8091/pools/default error 404 reason: unknown "unknown pool"
http://10.1.3.87:8091/pools/default with status False: unknown pool
2013-06-06 08:13:52 | ERROR | MainProcess | MainThread | [rest_client._http_request] http://10.1.3.87:8091/pools/default error 404 reason: unknown "unknown pool"
ERROR
======================================================================
ERROR: test_load (rebalancetests.IncrementalRebalanceInTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "pytests/rebalancetests.py", line 469, in tearDown
RebalanceBaseTest.common_tearDown(self)
File "pytests/rebalancetests.py", line 82, in common_tearDown
RebalanceBaseTest.reset(self)
File "pytests/rebalancetests.py", line 95, in reset
ClusterOperationHelper.cleanup_cluster([server])
File "lib/membase/helper/cluster_helper.py", line 269, in cleanup_cluster
rest = RestConnection(removed)
File "lib/membase/api/rest_client.py", line 281, in _init_
raise Exception("couchApiBase doesn't exist in nodes/self: %s " % http_res)
Exception: couchApiBase doesn't exist in nodes/self: {u'status': u'healthy', u'uptime': u'74131', u'clusterMembership': u'active', u'otpCookie': u'kydnbwvlxyqcmxtb', u'memoryQuota': 2457, u'memoryFree': 843325440, u'memoryTotal': 4294500352, u'hostname': u'10.1.3.87:8091', u'storage': {u'ssd': [], u'hdd': [
]}, u'storageTotals': {u'hdd':
{u'free': 34731477320, u'total': 64317550592, u'quotaTotal': 64317550592, u'usedByData': 0, u'used': 29586073272}, u'ram': {u'used': 3451174912, u'total': 4294500352, u'quotaTotal': 2576351232, u'usedByData': 0, u'quotaUsed': 0}}, u'thisNode': True, u'mcdMemoryReserved': 3276, u'clusterCompatibility': 1, u'version': u'2.1.0-701-rel-enterprise', u'otpNode': u'ns_1@10.1.3.87', u'mcdMemoryAllocated': 3276, u'availableStorage': {u'hdd': [
{u'usagePercent': 46, u'path': u'C:\\', u'sizeKBytes': 62810108}]}, u'systemStats':
{u'swap_total': 10882527232, u'swap_used': 3819888640, u'cpu_utilization_rate': 2.7060270602706029}, u'os': u'windows', u'ports':
{u'proxy': 11211, u'direct': 11210}, u'interestingStats': {}}