Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-2456

Transient NodeNotAvailableException rebalance under kv load

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 3.0.0
    • 3.0.1
    • library
    • None
    • 1

    Description

      I've been running a quite heavy kv workload and rebalancing kv nodes in and out (using 3.0.0). Not always, but sometimes I see an exception like:

      Couchbase.NodeNotAvailableException: Cannot find a Couchbase Server node for 10.143.194.101:11210.
         at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationToken token, Nullable`1 timeout)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at sample.Program.Main(String[] args) in /Users/michaelnitschinger/couchbase/code/dotnet/sample/Program.cs:line 19
      -----------------------Context Info---------------------------
      null
      

      So it seems that the SDK is not handling a state transition when nodes come and go properly (i.e. grabbing a new config and rescheduling the op transparently). Interestingly in the case above, the IP address was not one that I actually added or removed (10.143.194.101 was steady, in this case I added 10.143.194.102 in, but I've also seen it on rebalance out).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jmorris Jeff Morris added a comment -

          I couldn't reproduce this as specified. It may have been resolved by a commit on top of 3.0.0; if it comes back we can reopen the ticket and look deeper into a resolution.

          jmorris Jeff Morris added a comment - I couldn't reproduce this as specified. It may have been resolved by a commit on top of 3.0.0; if it comes back we can reopen the ticket and look deeper into a resolution.

          Code was rather trivial:

                  static async Task Main(string[] args)
                  {
                      var cluster = await Cluster.ConnectAsync("couchbase://10.143.194.101", "Administrator", "password");
                      var bucket = await cluster.BucketAsync("travel-sample");
                      var collection = bucket.DefaultCollection();
           
                      while (true) {
                          for (int i = 0; i < 1024; i++) {
                              try {
                              var upsertResult = await collection.UpsertAsync("doc1" + i, new {Name = "Ted", Age = 31});
                              var getResult = await collection.GetAsync("doc1" + i);
                              } catch (Exception e) {
                                  Console.WriteLine(e);
                              }
                              // Console.WriteLine(getResult.Cas);
                          }
                      }
           
                  }
          

          daschl Michael Nitschinger added a comment - Code was rather trivial: static async Task Main(string[] args) { var cluster = await Cluster.ConnectAsync("couchbase://10.143.194.101", "Administrator", "password"); var bucket = await cluster.BucketAsync("travel-sample"); var collection = bucket.DefaultCollection();   while (true) { for (int i = 0; i < 1024; i++) { try { var upsertResult = await collection.UpsertAsync("doc1" + i, new {Name = "Ted", Age = 31}); var getResult = await collection.GetAsync("doc1" + i); } catch (Exception e) { Console.WriteLine(e); } // Console.WriteLine(getResult.Cas); } }   }

          People

            jmorris Jeff Morris
            daschl Michael Nitschinger
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty