Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.5.1
-
Untriaged
-
-
Yes
Description
Removed a node and could not add back to cluster. Memcached Logs show that the node is waiting for a bucket delete to finish. The task gets down to 8 clients waiting. So far it's been in state for 3 hours.
2016-09-15T08:41:30.690285-07:00 NOTICE 257 The connected bucket is being deleted.. disconnecting
|
2016-09-15T08:41:30.692749-07:00 NOTICE 262 The connected bucket is being deleted.. disconnecting
|
2016-09-15T08:41:30.692715-07:00 NOTICE 253 Delete bucket [default]. Wait for clients to disconnect
|
2016-09-15T08:41:30.692864-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 58 clients connected
|
2016-09-15T08:41:31.694093-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 8 clients connected
|
...
|
2016-09-15T11:21:03.780160-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 8 clients connected
|
2016-09-15T11:21:04.780107-07:00 NOTICE 3930456088 Delete bucket [default]. Still waiting: 8 clients connected
|
|
ns_server logs 1 minute into waiting for bucket delete
[ns_server:error,2016-09-15T08:42:00.147-07:00,ns_1@172.23.106.213:<0.4113.982>:timeout_diag_logger:do_diag:118]Got timeout {slow_bucket_stop,{{single_bucket_kv_sup,"default"},
|
<0.21160.970>,supervisor,
|
[single_bucket_kv_sup]}}
|
Processes snapshot is:
|
|
[ns_server:error,2016-09-15T08:42:00.147-07:00,ns_1@172.23.106.213:<0.4113.982>:timeout_diag_logger:do_diag:120]
|
{<0.31114.971>,
|
[{registered_name,[]},
|
{status,waiting},
|
{initial_call,{inet_tcp_dist,do_setup,6}},
|
{backtrace,[<<"Program counter: 0x00007fd27515d330 (dist_util:con_loop/9 + 112)">>,
|
<<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>,
|
<<"0x00007fd22f60d3b0 Return addr 0x0000000000893588 (<terminate process normally>)">>,
|
<<"y(0) []">>,<<"y(1) #Fun<inet_tcp_dist.getstat.1>">>,
|
<<"y(2) #Fun<inet_tcp_dist.tick.1>">>,
|
<<"y(3) {tick,7430,7436,1,1}">>,<<"y(4) normal">>,
|
<<"y(5) 'ns_1@172.23.106.213'">>,
|
|
node is currently online in this state: 172.23.106.213
https://s3.amazonaws.com/scalability-mcafee/collectinfo-2016-09-15T134105-ns_1%40172.23.106.213.zip
~6 days into longevity test
Attachments
Issue Links
- blocks
-
MB-19612 4.5.1 Minor Release
- Closed