Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35407

graceful failover deletes bucket with force=true and can lead to poor delta recovery

    XMLWordPrintable

Details

    Description

      The RCA for MB-35326 showed that a dead->replica state change occurred soon before the bucket was deleted, the replica state change was never flushed because a force delete was used.

      see comment here

      The result of this is that KV at the restart/bucket creation, warmup will ignore dead vbuckets and as such, the vbuckets which could of been warmed up (because they were correctly flushed as active or replica) must now be rebuilt.

      Raising this MB as it seems a force bucket delete seems to be counter productive for a graceful failover->delta recovery workflow.

      Note-1: force shutdown noticed in ns_server.debug.log

      [ns_server:info,2019-07-27T23:34:56.175-07:00,ns_1@172.23.105.47:ns_memcached-default<0.20811.9>:ns_memcached:delete_bucket:770]Deleting bucket "default" from memcached (force = true)
      

       
      Note-2: Assuming graceful failover as it's mentioned in logging e.g.

      [ns_server:debug,2019-07-27T23:34:56.831-07:00,ns_1@172.23.105.47:ns_config_log<0.199.0>:ns_config_log:log_common:231]config change:
      counters ->
      [{'_vclock',[{<<"66664331ef97ea7bd2fdcdbe29e89002">>,{54,63731514896}}]},
       {failover_complete,2},
       {graceful_failover_start,2},
       {rebalance_success,24},
       {rebalance_start,24},
       {graceful_failover_success,1},
       {failover,1}]
      

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            jwalker Jim Walker
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty