Note-2: Assuming graceful failover as it's mentioned in logging e.g.

Yup - slightly more explicit reference in diag

2019-07-27T23:34:17.505-07:00, ns_orchestrator:0:info:message(ns_1@172.23.105.105) - Starting graceful failover of nodes ['ns_1@172.23.105.47']. Operation Id = a5447c50d1e82b8739d0dd3efd101965

2019-07-27T23:34:17.506-07:00, ns_rebalancer:0:info:message(ns_1@172.23.105.105) - Starting vbucket moves for graceful failover of ['ns_1@172.23.105.47']

2019-07-27T23:34:18.081-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.105.105) - Bucket "default" rebalance does not seem to be swap rebalance

2019-07-27T23:34:55.796-07:00, failover:0:info:message(ns_1@172.23.105.105) - Starting failing over ['ns_1@172.23.105.47']

2019-07-27T23:34:56.175-07:00, ns_memcached:0:info:message(ns_1@172.23.105.47) - Shutting down bucket "default" on 'ns_1@172.23.105.47' for deletion

2019-07-27T23:34:56.808-07:00, failover:0:info:message(ns_1@172.23.105.105) - Failed over ['ns_1@172.23.105.47']: ok

2019-07-27T23:34:56.838-07:00, failover:0:info:message(ns_1@172.23.105.105) - Deactivating failed over nodes ['ns_1@172.23.105.47']

2019-07-27T23:34:56.842-07:00, ns_orchestrator:0:info:message(ns_1@172.23.105.105) - Graceful failover completed successfully.

When ns_memcached:do_terminate is called with Reason other than

occur are when either:

handle_cast({connect_done... does an ensure_bucket and gets an exception from memcached_bucket_config:get when it can't get the bucket info

or

do_ensure_bucket gets "restart" back from memcached_bucket_config:ensure which happens if any param has changed and has [restart] property (see memcached_bucket_config:arams(membase...

I'll have to consult with Aliaksey Artamonau to get his thoughts on this.

The result of this is that KV at the restart/bucket creation, warmup will ignore dead vbuckets and as such, the vbuckets which could of been warmed up (because they were correctly flushed as active or replica) must now be rebuilt.

Is this a new behavior for memcached to not warmup dead vbuckets? What will happen when ns_server then changes the vbucket state to either active or replica? Will the vbucket wind up being empty? If so, then this behavior is unacceptable because it may arise in other scenarios.

Reading the linked ticket, and it appears that the answer to my question is: yes, a vbucket that was dead at warmup time will lose its data even if turned into an active or a replica vbucket. I think this is a problem and needs to be fixed. Otherwise in certain cases we will lose data when we are not supposed to.

As to the issue of bucket deletion after delta recovery. Yes we could do better. Unfortunately the code is written in such a way that it'll be hard to change the way buckets are deleted. But it should be relatively easy to add an extra step to the delta recovery rebalance that will wait till all the data is persisted on the delta nodes. Is it worth the effort? Not sure.

Aliaksey Artamonau

Is this a new behavior for memcached to not warmup dead vbuckets?

3 years old, since 5.0 no MB or significant background to the change https://github.com/couchbase/kv_engine/commit/b379c2525d9b91bdf441c350d1b739fcd6ebcb22

Will the vbucket wind up being empty?

Yes, we will start up and have no vbucket at all for any of the dead ones, ns_server then corrects KV and a new empty vbucket will be created.

In the discussions of MB-35326 it was a surprise that this happens, I certainly concluded that a VB state transition active->dead->replica with no crash should be logically equivalent to active->dead->warmup->replica, no loss of data from the node.

Dave Rigby I vote that we effectively revert https://github.com/couchbase/kv_engine/commit/b379c2525d9b91bdf441c350d1b739fcd6ebcb22

Dave Rigby I vote that we effectively revert https://github.com/couchbase/kv_engine/commit/b379c2525d9b91bdf441c350d1b739fcd6ebcb22

SGTM.

Although I do think that ns_server should be more careful with controlled shutdown - force=true won't wait for outstanding mutations etc to flush, so it can lead to data-loss (assuming durability isn't used).

Dave Rigby I agree, a graceful failover with a non-graceful shutdown still seems like a legit problem.

I'll move the warmup revert to a separate MB and this one can track the non-graceful issue - back to ns_server for this MB.

• KV warmup issue tracked as https://issues.couchbase.com/browse/MB-35599

We only use force=true for failover (graceful or otherwise). So I don't think it's a critical issue: the recovery should ensure that no data (in case of graceful failover) is actually lost.