Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-1729

Operation timeout when server(spock) restarts

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.1
    • Component/s: None
    • Labels:
      None

      Description

      with commit f1e01cb24eab23fe963186141aae4d1a27f8a11e which is top of master as of 7/9 and candidate for version 2.6.0, when all Spock(5.0) servers are restarting one by one, couple of timeout are detected about 70 seconds after servers are started (servers started means service start couchbase command placed, so it would take some time to actually server is up and running since then)

      Attaching SDK log and graph link is http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-5.0.0-xxxx/SvcRestartAll-HYBRID/07-09-18/021208/2ea9aab5aca092bd8b571a0e6e2ef9be-MC.html

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          jmorris Jeff Morris added a comment - - edited

          Each timeout corresponds to a send timeout - the request is sent but the response is not received within the ClientConfiguration.SendTimeout of 15s:

          2018-07-09 22:56:20,978 [68] DEBUG Couchbase.Core.Server - Sending Set`1 with key SimpleKeyREP3930REP3930REP3930REP3930REP3930 using server 172.23.109.150:11210
          2018-07-09 22:56:20,978 [44] DEBUG Couchbase.IO.Services.PooledIOService - Couchbase.IO.SendTimeoutExpiredException: The operation has timed out. {"s":"kv","i":"e8c8","c":"0a1ef0fb9d86f17c/8621a1b4d93e85ae","b":"default","l":"172.23.109.139:51255","r":"172.23.109.142:11210","t":15000000}
           at Couchbase.IO.MultiplexingConnection.Send(Byte[] request)
           at Couchbase.IO.Services.PooledIOService.Execute[T](IOperation`1 operation)
          2018-07-09 22:56:20,978 [44] INFO Couchbase.Core.Server - Checking if node 172.23.109.142:11210 should be down - last: 22:54:28.5243695, current: 22:56:20.9785403, count: 3
          2018-07-09 22:56:20,978 [44] DEBUG Couchbase.Core.Buckets.CouchbaseRequestExecuter - Operation doesn't support retries for key SimpleKeyREP4970REP4970REP4970REP4970REP4970
          2018-07-09 22:56:20,978 [44] DEBUG Couchbase.Core.Buckets.RequestExecuterBase - Operation for key SimpleKeyREP4970REP4970REP4970REP4970REP4970 failed after 1 retries using vb0 from rev43 and opaque59592. Reason: The operation has timed out.

          Show
          jmorris Jeff Morris added a comment - - edited Each timeout corresponds to a send timeout - the request is sent but the response is not received within the ClientConfiguration.SendTimeout of 15s: 2018 - 07 - 09 22 : 56 : 20 , 978 [ 68 ] DEBUG Couchbase.Core.Server - Sending Set` 1 with key SimpleKeyREP3930REP3930REP3930REP3930REP3930 using server 172.23 . 109.150 : 11210 2018 - 07 - 09 22 : 56 : 20 , 978 [ 44 ] DEBUG Couchbase.IO.Services.PooledIOService - Couchbase.IO.SendTimeoutExpiredException: The operation has timed out. { "s" : "kv" , "i" : "e8c8" , "c" : "0a1ef0fb9d86f17c/8621a1b4d93e85ae" , "b" : "default" , "l" : "172.23.109.139:51255" , "r" : "172.23.109.142:11210" , "t" : 15000000 } at Couchbase.IO.MultiplexingConnection.Send(Byte[] request) at Couchbase.IO.Services.PooledIOService.Execute[T](IOperation` 1 operation) 2018 - 07 - 09 22 : 56 : 20 , 978 [ 44 ] INFO Couchbase.Core.Server - Checking if node 172.23 . 109.142 : 11210 should be down - last: 22 : 54 : 28.5243695 , current: 22 : 56 : 20.9785403 , count: 3 2018 - 07 - 09 22 : 56 : 20 , 978 [ 44 ] DEBUG Couchbase.Core.Buckets.CouchbaseRequestExecuter - Operation doesn't support retries for key SimpleKeyREP4970REP4970REP4970REP4970REP4970 2018 - 07 - 09 22 : 56 : 20 , 978 [ 44 ] DEBUG Couchbase.Core.Buckets.RequestExecuterBase - Operation for key SimpleKeyREP4970REP4970REP4970REP4970REP4970 failed after 1 retries using vb0 from rev43 and opaque59592. Reason: The operation has timed out.
          Hide
          jmorris Jeff Morris added a comment - - edited

          Jae Park [X] -

          Based on my last comment, it looks like the server isn't responding in a timely manner and then the client times out the operation. We might want to do a TCP dump and/or correlate with server logs.

          Jeff

          Show
          jmorris Jeff Morris added a comment - - edited Jae Park [X] - Based on my last comment, it looks like the server isn't responding in a timely manner and then the client times out the operation. We might want to do a TCP dump and/or correlate with server logs. Jeff
          Hide
          jmorris Jeff Morris added a comment -

          Moving this up a release - need more time and info from QE

          Show
          jmorris Jeff Morris added a comment - Moving this up a release - need more time and info from QE
          Hide
          jmorris Jeff Morris added a comment -

          Jae Park [X] - can you verify that this is resolved?

          Show
          jmorris Jeff Morris added a comment - Jae Park [X]  - can you verify that this is resolved?
          Hide
          jaekwon.park Jae Park [X] (Inactive) added a comment -

          verified 2.7.0

          Show
          jaekwon.park Jae Park [X] (Inactive) added a comment - verified 2.7.0

            People

            • Assignee:
              jaekwon.park Jae Park [X] (Inactive)
              Reporter:
              jaekwon.park Jae Park [X] (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.