Uploaded image for project: 'Couchbase Go SDK'
  1. Couchbase Go SDK
  2. GOCBC-1301

ResumeTransactionAttempt does not set addLostCleanupLocation

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.5.1, core-10.1.3
    • None
    • None
    • None
    • 1
    • SDK24: Lambda + Known Issues

    Description

      When running query transactions in our automated transaction tests the service appears to be crashing and then coming back up.

      The output we are receiving is:

      Transaction has failed with cause 'com.couchbase.client.core.error.RequestCanceledException: QueryRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)

      This seems to be happening in the seperate Java transaction library, the integrated Java library and the Go library, failing even basic tests.

      More detailed output logs from the Java performer:

      14:59:03.254 [grpc-default-executor-0] INFO  c.c.PerformerTransactionService - ================ ensureUpsertWorks() : testStart ================ 
      14:59:03.472 [grpc-default-executor-0] INFO  c.c.PerformerTransactionService - Creating new transaction with cluster connection a9cfd53b-8315-484d-b46a-a160452f01b6 (user Administrator) and transactions factory c81bb5c4-46f2-421e-b667-75377a933dc0
      		 
      14:59:03.520 [cb-txn-5] INFO   - Starting attempt 0 5eefe185-0a89-40bc-afb6-b12c0fe74c47
      		 
      14:59:03.520 [cb-txn-5] INFO   - Running command 'Query UPSERT INTO `default` VALUES ('b14be9ed-9ce3-4543-a517-0958d76d15f8', {"content":"initial"})'
      		 
      14:59:04.292 [cb-txn-5] INFO   - Took 772 millis to run command 'Query UPSERT INTO `default` VALUES ('b14be9ed-9ce3-4543-a517-0958d76d15f8', {"content":"initial"})'
      		 
      14:59:04.293 [cb-txn-5] INFO   - Reached end of all operations and lambda
      		 
      14:59:04.527 [grpc-default-executor-0] INFO   - Error while executing an operation: com.couchbase.transactions.error.TransactionFailed: Transaction has failed with cause 'com.couchbase.client.core.error.RequestCanceledException: QueryRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) {"cancelled":true,"completed":true,"coreId":"0xfcc9553f00000001","idempotent":false,"lastDispatchedFrom":"192.168.100.250:51728","lastDispatchedTo":"172.23.111.135:8093","reason":"NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)","requestId":55,"requestType":"QueryRequest","retried":0,"service":{"operationId":"7240fa7b-5398-4829-813b-a4e4d9b588dc","statement":"<ud>COMMIT</ud>","target":"NodeIdentifier{address=172.23.111.135, managerPort=8091}","type":"query"},"timeoutMs":25184}'
      		 
      14:59:04.528 [grpc-default-executor-0] INFO   - Waiting for all expected events
      		 
      14:59:04.549 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"circuitBreaker":"DISABLED","connectedForMs":794,"coreId":"0xfcc9553f00000001","local":"192.168.100.250:51727","numOutstandingRequests":0,"remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      14:59:04.549 [cb-events] INFO  com.couchbase.request - [com.couchbase.request][RequestNotRetriedEvent] Request QueryRequest not retried per RetryStrategy (Reason: CHANNEL_CLOSED_WHILE_IN_FLIGHT) {"cancelled":true,"completed":true,"coreId":"0xfcc9553f00000001","idempotent":false,"lastDispatchedFrom":"192.168.100.250:51728","lastDispatchedTo":"172.23.111.135:8093","reason":"NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)","requestId":55,"requestType":"QueryRequest","retried":0,"service":{"operationId":"7240fa7b-5398-4829-813b-a4e4d9b588dc","statement":"<ud>COMMIT</ud>","target":"NodeIdentifier{address=172.23.111.135, managerPort=8091}","type":"query"},"timeoutMs":25184,"timings":{"totalMicros":231218}}
      		 
      14:59:04.549 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"circuitBreaker":"DISABLED","connectedForMs":470,"coreId":"0xfcc9553f00000001","local":"192.168.100.250:51728","numOutstandingRequests":1,"remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      14:59:04.713 [grpc-default-executor-0] INFO  c.c.PerformerTransactionService - ================ ensureUpsertWorks() : testEnd ================ 
      14:59:04.759 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][160ms] Connect attempt 1 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51727","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:04.760 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][156ms] Connect attempt 1 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51728","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:04.968 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][168ms] Connect attempt 2 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51727","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:04.969 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][160ms] Connect attempt 2 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51728","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:05.177 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][160ms] Connect attempt 3 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51727","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:05.177 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][156ms] Connect attempt 3 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51728","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:05.487 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][157ms] Connect attempt 4 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51727","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:05.487 [cb-events] WARN  com.couchbase.endpoint - [com.couchbase.endpoint][EndpointConnectionFailedEvent][157ms] Connect attempt 4 failed because of : finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting. {"circuitBreaker":"DISABLED","coreId":"0xfcc9553f00000001","local":"192.168.100.250:51728","remote":"172.23.111.135:8093","type":"QUERY"}
      		 
      com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.111.135:8093 - Check server ports and cluster encryption setting.
      		 
      14:59:05.658 [grpc-default-executor-0] INFO  c.c.PerformerTransactionService - ================ Test file : End ================

      As you can see we get the 'out of retries' output followed by a loss of connection to the n1ql port.

      Minimum reproduction in Java SDK:

      Cluster cluster = Cluster.connect("172.23.111.135", "Administrator", "password");
      		 
      cluster.transactions().run((ctx) -> {
      		 
          ctx.query("INSERT INTO `default` VALUES (\"doc1\",{\"content\":\"initial\"})");
      		 
      });

      When running the exact statement in the UI and in TX mode the statement works fine.

      As mentioned the tests failing (2-300) are all passing for 7.1 and 7.0 versions with 0 code change and the same issues are being seen across multiple SDKs which points to some common issue in the server.

      Server build: Enterprise Edition 7.2.0 build 1312

      CC: Graham Pople Praneeth Bokka Arunkumar Senthilnathan 

      Attachments

        Issue Links

          mentioned in

          Page Loading...

          For Gerrit Dashboard: GOCBC-1301
          # Subject Branch Project Status CR V
          176237,3 GOCBC-1301: Set addLostCleanupLocation in ResumeTransactionAttempt master gocbcore Status: MERGED +2 +1

          Activity

            People

              charles.dixon Charles Dixon
              matthew.bray Matthew Bray
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty