Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-1393

async upsert hangs after removing entry point node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.4.5
    • None
    • None
    • None

    Description

      client version : couchbase-net-client 2.4.2

      server version : 4.5.1 GA

      Reproduction steps:

      1. run 2 or more cluster
      2. open bucket using one of the node host (entry point)
      3. keep upsert.
      4. while upsert, failover the entry point and leave it for more than 15 seconds.

      Expected : upsert should work with active nodes

      Current : upsert hangs

       

      Details : net client waits for lock which is not released by previous call to 

      https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase/Core/Server.cs#L414

       

      Here are two call stacks happened exactly 15 second after the other lock

      First lock right after node is failed over:
      >    Couchbase.NetClient.dll!Couchbase.Core.Server.CheckOnline(bool isDead) Line 416    C#
          Couchbase.NetClient.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.MultiplexingConnection>.Release(Couchbase.IO.MultiplexingConnection connection) Line 214    C#
          Couchbase.NetClient.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.MultiplexingConnection>.Couchbase.IO.IConnectionPool.Release(Couchbase.IO.IConnection connection) Line 315    C#
          Couchbase.NetClient.dll!Couchbase.IO.Services.MultiplexingIOService.CheckConnection() Line 378    C#
          Couchbase.NetClient.dll!Couchbase.IO.Services.MultiplexingIOService.ExecuteAsync<object>(Couchbase.IO.Operations.IOperation<object> operation, Couchbase.IO.IConnection connection) Line 197    C#
          Couchbase.NetClient.dll!Couchbase.IO.Services.MultiplexingIOService.ExecuteAsync<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 209    C#
          Couchbase.NetClient.dll!Couchbase.Core.Server.SendAsync<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 632    C#
          Couchbase.NetClient.dll!Couchbase.Core.Buckets.CouchbaseRequestExecuter.SendWithRetryAsync<object>(Couchbase.IO.Operations.IOperation<object> operation, System.Threading.Tasks.TaskCompletionSource<Couchbase.IOperationResult<object>> tcs, System.Threading.CancellationTokenSource cts) Line 740    C#
          Couchbase.NetClient.dll!Couchbase.Core.Buckets.CouchbaseRequestExecuter.SendWithDurabilityAsync<object>(Couchbase.IO.Operations.IOperation<object> operation, bool deletion, Couchbase.ReplicateTo replicateTo, Couchbase.PersistTo persistTo) Line 345    C#
          Couchbase.NetClient.dll!Couchbase.CouchbaseBucket.UpsertAsync<object>(string key, object value, ulong cas, uint expiration, Couchbase.ReplicateTo replicateTo, Couchbase.PersistTo persistTo) Line 3412    C#
          Couchbase.NetClient.dll!Couchbase.CouchbaseBucket.UpsertAsync<object>(string key, object value, Couchbase.ReplicateTo replicateTo, Couchbase.PersistTo persistTo) Line 3362    C#
          Sdkd.dll!Sdkd.Commands.SetCommand.ExecuteSingleAsync(System.Collections.Generic.Dictionary<string, string> batch) Line 67    C#
          Sdkd.dll!Sdkd.Commands.KVCommand.Run(int handleId) Line 42    C#
          SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.IBucket cli) Line 271    C#
          SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 141    C#
       
       
      Second lock after 15 seconds :
      Couchbase.NetClient.dll!Couchbase.Core.Server.CheckOnline(bool isDead) Line 416    C#
          Couchbase.NetClient.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.MultiplexingConnection>.Release(Couchbase.IO.MultiplexingConnection connection) Line 214    C#
          Couchbase.NetClient.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.MultiplexingConnection>.Couchbase.IO.IConnectionPool.Release(Couchbase.IO.IConnection connection) Line 315    C#
          Couchbase.NetClient.dll!Couchbase.IO.Services.MultiplexingIOService.CheckConnection() Line 393    C#
          Couchbase.NetClient.dll!Couchbase.IO.Services.MultiplexingIOService.ExecuteAsync<object>(Couchbase.IO.Operations.IOperation<object> operation, Couchbase.IO.IConnection connection) Line 197    C#
          Couchbase.NetClient.dll!Couchbase.IO.Services.MultiplexingIOService.ExecuteAsync<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 209    C#
          Couchbase.NetClient.dll!Couchbase.Core.Server.SendAsync<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 632    C#
          Couchbase.NetClient.dll!Couchbase.Core.Buckets.CouchbaseRequestExecuter.SendWithRetryAsync<object>(Couchbase.IO.Operations.IOperation<object> operation, System.Threading.Tasks.TaskCompletionSource<Couchbase.IOperationResult<object>> tcs, System.Threading.CancellationTokenSource cts) Line 740    C#
          Couchbase.NetClient.dll!Couchbase.Core.Buckets.CouchbaseRequestExecuter.SendWithDurabilityAsync<object>(Couchbase.IO.Operations.IOperation<object> operation, bool deletion, Couchbase.ReplicateTo replicateTo, Couchbase.PersistTo persistTo) Line 345    C#
          Couchbase.NetClient.dll!Couchbase.CouchbaseBucket.UpsertAsync<object>(string key, object value, ulong cas, uint expiration, Couchbase.ReplicateTo replicateTo, Couchbase.PersistTo persistTo) Line 3412    C#
          Couchbase.NetClient.dll!Couchbase.CouchbaseBucket.UpsertAsync<object>(string key, object value, Couchbase.ReplicateTo replicateTo, Couchbase.PersistTo persistTo) Line 3362    C#
          Sdkd.dll!Sdkd.Commands.SetCommand.ExecuteSingleAsync(System.Collections.Generic.Dictionary<string, string> batch) Line 67    C#
          Sdkd.dll!Sdkd.Commands.KVCommand.Run(int handleId) Line 42    C#
          SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.IBucket cli) Line 271    C#
          SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 141    C#
       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jmorris Jeff Morris
              jaekwon.park Jae Park [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty