Description
obvious from these logs :
Rebalance exited with reason
(repeated 1 times) ns_orchestrator002 ns_1@10.113.11.11 01:24:57 - Mon Jun 4, 2012
Rebalance exited with reason {wait_for_memcached_failed,"Authentication", ['ns_1@10.113.11.5','ns_1@10.113.11.11', 'ns_1@10.113.11.12','ns_1@10.113.11.16', 'ns_1@10.113.11.17','ns_1@10.113.11.19', 'ns_1@10.113.11.21','ns_1@10.113.11.23', 'ns_1@10.113.11.24','ns_1@10.113.11.27']}
ns_orchestrator002 ns_1@10.113.11.11 01:19:18 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
ns_orchestrator004 ns_1@10.113.11.11 01:19:07 - Mon Jun 4, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.113.11.11 01:13:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.17' for server shutdown ns_memcached002 ns_1@10.113.11.17 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.19' for server shutdown ns_memcached002 ns_1@10.113.11.19 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.23' for server shutdown ns_memcached002 ns_1@10.113.11.23 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.11' for server shutdown ns_memcached002 ns_1@10.113.11.11 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.12' for server shutdown ns_memcached002 ns_1@10.113.11.12 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.5' for server shutdown ns_memcached002 ns_1@10.113.11.5 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.16' for server shutdown ns_memcached002 ns_1@10.113.11.16 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.24' for server shutdown ns_memcached002 ns_1@10.113.11.24 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.21' for server shutdown ns_memcached002 ns_1@10.113.11.21 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.27' for server shutdown ns_memcached002 ns_1@10.113.11.27 01:02:19 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
Attachments
For Gerrit Dashboard: MB-5434 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
16783,2 | MB-5434: don't add ejected nodes back to bucket's servers list | branch-181 | ns_server | Status: MERGED | +2 | +1 |
16829,1 | Merge remote-tracking branch 'origin/branch-181' into branch-18 | branch-18 | ns_server | Status: MERGED | +2 | +1 |
16848,1 | Merge branch 'branch-18' | master | ns_server | Status: MERGED | +2 | +1 |