Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
master, 6.5.0
Description
The RCA for MB-35326 showed that a dead->replica state change occurred soon before the bucket was deleted, the replica state change was never flushed because a force delete was used.
The result of this is that KV at the restart/bucket creation, warmup will ignore dead vbuckets and as such, the vbuckets which could of been warmed up (because they were correctly flushed as active or replica) must now be rebuilt.
Raising this MB as it seems a force bucket delete seems to be counter productive for a graceful failover->delta recovery workflow.
Note-1: force shutdown noticed in ns_server.debug.log
[ns_server:info,2019-07-27T23:34:56.175-07:00,ns_1@172.23.105.47:ns_memcached-default<0.20811.9>:ns_memcached:delete_bucket:770]Deleting bucket "default" from memcached (force = true)
|
Note-2: Assuming graceful failover as it's mentioned in logging e.g.
[ns_server:debug,2019-07-27T23:34:56.831-07:00,ns_1@172.23.105.47:ns_config_log<0.199.0>:ns_config_log:log_common:231]config change:
|
counters ->
|
[{'_vclock',[{<<"66664331ef97ea7bd2fdcdbe29e89002">>,{54,63731514896}}]},
|
{failover_complete,2},
|
{graceful_failover_start,2},
|
{rebalance_success,24},
|
{rebalance_start,24},
|
{graceful_failover_success,1},
|
{failover,1}]
|
Yup - slightly more explicit reference in diag
2019-07-27T23:34:17.505-07:00, ns_orchestrator:0:info:message(ns_1@172.23.105.105) - Starting graceful failover of nodes ['ns_1@172.23.105.47']. Operation Id = a5447c50d1e82b8739d0dd3efd101965
2019-07-27T23:34:17.506-07:00, ns_rebalancer:0:info:message(ns_1@172.23.105.105) - Starting vbucket moves for graceful failover of ['ns_1@172.23.105.47']
2019-07-27T23:34:18.081-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.105.105) - Bucket "default" rebalance does not seem to be swap rebalance
2019-07-27T23:34:55.796-07:00, failover:0:info:message(ns_1@172.23.105.105) - Starting failing over ['ns_1@172.23.105.47']
2019-07-27T23:34:56.175-07:00, ns_memcached:0:info:message(ns_1@172.23.105.47) - Shutting down bucket "default" on 'ns_1@172.23.105.47' for deletion
2019-07-27T23:34:56.808-07:00, failover:0:info:message(ns_1@172.23.105.105) - Failed over ['ns_1@172.23.105.47']: ok
2019-07-27T23:34:56.838-07:00, failover:0:info:message(ns_1@172.23.105.105) - Deactivating failed over nodes ['ns_1@172.23.105.47']
2019-07-27T23:34:56.842-07:00, ns_orchestrator:0:info:message(ns_1@172.23.105.105) - Graceful failover completed successfully.