Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41412

Durability level=majority should work in a single node cluster

    XMLWordPrintable

Details

    • 1

    Description

      If you do a durable write in a single node cluster, a 'durability impossible' exception gets thrown back.

      This is because by default the number of replicas for a new bucket = 1. Hence, in a single node cluster it is impossible to achieve majority (which is 2 nodes if you have 1 replica).

      This is undesirable out-of-box experience as many developers will play with single node clusters.

      In discussion with Dave Rigby here are some of the things he suggested:
      In terms of improving this experience, I see two approaches:
       
      1. Change the default number of replicas in the “Create Bucket” to zero (perhaps only if there’s one node currently in the cluster?).

      2. Have ns_server somehow modify the chain it presents to KV-Engine - maybe capping the number of elements in the chain based on the total node count (i.e. until you add more than 1 node the chain just has one element).
       __ 

      Option (2) could be interesting - given you’d have to rebalance anyway if you wanted to add a second node and start using it, having the chain smaller until the rebalance occurs would I think address this (and similar issues when changing replica counts to >1).

      The problem I see with Option 1 is (as pointed out by DaveR again) is that every cluster will start out as a single node cluster (users will add nodes to a single node). Hence, existing users will be thrown off if we change the default replicas to zero (as well as end up with unsafe clusters). Hence Option 2 may be the only real choice.

      Filing it under KV, but probably involves ns_server work too.

      Dave Finlay FYI

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Dave Rigby how should we determine the next step for this? 

          shivani.gupta Shivani Gupta added a comment - Dave Rigby  how should we determine the next step for this? 
          drigby Dave Rigby added a comment -

          Shivani Gupta I think we need to reach agreement with ns_server on what approach we should take to solve this - the vbucket topology is owned by ns_sever, KV-Engine is simply informed when it changes.

          drigby Dave Rigby added a comment - Shivani Gupta I think we need to reach agreement with ns_server on what approach we should take to solve this - the vbucket topology is owned by ns_sever, KV-Engine is simply informed when it changes.
          dfinlay Dave Finlay added a comment - - edited

          Thanks all. Interesting request. Initially I though that the best way to tackle this problem would be to add a new property to the bucket configuration that captures the idea of the number of provisionable replicas of a bucket. (Perhaps there's a better name than "provisionable".) Every time we rebalance we would recompute num_provisionable_replicas as follows:

          num_provisionable_replicas = min(num_replicas, number-of-KV-nodes-in-cluster)
          

          And we'd build the vbucket map based on the provisionable replica count, not the configured replica count. But this is actually very close to what we do today (since we already build the vbucket map based on the number of nodes on which we can provision replicas, since we need a vbucket map that actually works even if it's constrained.)

          So, we may be able to not add a persisted notion of what the provisionable replica count is and just to drop the extra padded "undefined" replicas at the end of the replication chain and it might work. Bringing back to CC for now on this basis.

          dfinlay Dave Finlay added a comment - - edited Thanks all. Interesting request. Initially I though that the best way to tackle this problem would be to add a new property to the bucket configuration that captures the idea of the number of provisionable replicas of a bucket. (Perhaps there's a better name than "provisionable".) Every time we rebalance we would recompute num_provisionable_replicas as follows: num_provisionable_replicas = min(num_replicas, number-of-KV-nodes-in-cluster) And we'd build the vbucket map based on the provisionable replica count, not the configured replica count. But this is actually very close to what we do today (since we already build the vbucket map based on the number of nodes on which we can provision replicas, since we need a vbucket map that actually works even if it's constrained.) So, we may be able to not add a persisted notion of what the provisionable replica count is and just to drop the extra padded "undefined" replicas at the end of the replication chain and it might work. Bringing back to CC for now on this basis.

          Thanks Dave Finlay, that's welcome news!

          shivani.gupta Shivani Gupta added a comment - Thanks Dave Finlay , that's welcome news!
          drigby Dave Rigby added a comment -

          I've been reviewing the outstanding work for SyncWrites again.

          I think we should come back to one of the "simpler" original suggestions - perhaps we change the default number of replicas to be 0 iff the number of nodes in the cluster is 1; else it stays at 1.

          This could arguably be considered "surprising" to users - given they could end up with a different number of replicas than what they used to (say if they created a bucket on a single-node cluster first; and then added additional nodes), but pragmatically I think it might be a reasonable approach - and we address with documentation / UI hints.

          Unfortunately a major release like 7.0 would be the best time to make such a breaking change, but it's obviously far too late for that now.

          (Other options such as making the number of replicas explicit (i.e. it's a new mandatory parameter when creating a bucket) would at least be very explicit to the user, but could break existing applications/scripts creating Buckets programmatically via the REST API).

          drigby Dave Rigby added a comment - I've been reviewing the outstanding work for SyncWrites again. I think we should come back to one of the "simpler" original suggestions - perhaps we change the default number of replicas to be 0 iff the number of nodes in the cluster is 1; else it stays at 1. This could arguably be considered "surprising" to users - given they could end up with a different number of replicas than what they used to (say if they created a bucket on a single-node cluster first; and then added additional nodes), but pragmatically I think it might be a reasonable approach - and we address with documentation / UI hints. Unfortunately a major release like 7.0 would be the best time to make such a breaking change, but it's obviously far too late for that now. (Other options such as making the number of replicas explicit (i.e. it's a new mandatory parameter when creating a bucket) would at least be very explicit to the user, but could break existing applications/scripts creating Buckets programmatically via the REST API).

          People

            dfinlay Dave Finlay
            shivani.gupta Shivani Gupta
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty