Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34162

Transactions timed out in YCSB with latest 3.X SDK

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Transactions are timing out in YCSB on recent mad-hatter build 6.5.0-3216 while latest couchbase-transactions-1.0.0-alpha.3.jar dowloaded from (https://hub.internal.couchbase.com/confluence/pages/viewpage.action?spaceKey=SDK&title=Transactions ).

      Based on logs It looks like transactions are failing  while creating/retrieving Subdoc's . Not sure its a SDK/Server kv issue . Please take a look at the detailed logs attached 

       

      Please find implementation details and SDK/transaction logs (attached) .

       

      YCSB Implementation details in github - 

      https://github.com/couchbaselabs/YCSB/tree/couchbase3-transactions-playground

       

      Error Snippet -

       "Transaction logger:14/Thread-1/257f6fd1-75d8-4969-a40f-ddc2b839975c starting attempt 3/026fae55-b851-4ebd-86b7-65fbd04c1139

      Transaction logger:14/Thread-1/257f6fd1-75d8-4969-a40f-ddc2b839975c/026fae55-b851-4ebd-86b7-65fbd04c1139 getting doc usertable:user6873002678636213555

      _Transaction logger:14/Thread-1/257f6fd1-75d8-4969-a40f-ddc2b839975c/026fae55-b851-4ebd-86b7-65fbd04c1139 caught exception 'com.couchbase.client.core.error.RequestTimeoutException: SubdocGetRequest {"retried":13,"reason":"TIMEOUT","requestId":23,"timeoutMs":2500,"service":

      {"bucket":"bucket-1","type":"kv","key":"usertable:user6873002678636213555","cid":0}

      ,"cancelled":true,"coreId":1,"completed":true}' in asyncInternal, rethrowing to rollback_

      "

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Hi Graham - Please find attached transaction logs and cbcollect logs for analysis . 

          You can also run YCSB by following the build instructions on your local . Please replace the host ip as needed . 

          Command to load docs : 

          bin/ycsb load couchbase3 -P workloads/workloadta -p writeallfields=true -threads 1 -p couchbase.host=172.23.109.175 -p couchbase.bucket=bucket-1 -p couchbase.upsert=true -p couchbase.password=password -p operationcount=100 -p recordcount=100 -p couchbase.transactionsEnabled=false 

           

          Command to run transactions :

          bin/ycsb run couchbase3 -P workloads/workloadta -p writeallfields=true -threads 1 -p couchbase.host=172.23.109.175 -p couchbase.bucket=bucket-1 -p couchbase.upsert=true -p couchbase.password=password -p operationcount=100 -p recordcount=100 -p couchbase.transactionsEnabled=true > ycsb_transactions.log

           

          sharath.sulochana Sharath Sulochana (Inactive) added a comment - Hi Graham - Please find attached transaction logs and cbcollect logs for analysis .  You can also run YCSB by following the build instructions on your local . Please replace the host ip as needed .  Command to load docs :  bin/ycsb load couchbase3 -P workloads/workloadta -p writeallfields=true -threads 1 -p couchbase.host= 172.23.109.175 -p couchbase.bucket=bucket-1 -p couchbase.upsert=true -p couchbase.password=password -p operationcount=100 -p recordcount=100 -p couchbase.transactionsEnabled=false    Command to run transactions : bin/ycsb run couchbase3 -P workloads/workloadta -p writeallfields=true -threads 1 -p couchbase.host= 172.23.109.175 -p couchbase.bucket=bucket-1 -p couchbase.upsert=true -p couchbase.password=password -p operationcount=100 -p recordcount=100 -p couchbase.transactionsEnabled=true > ycsb_transactions.log  
          graham.pople Graham Pople added a comment -

          The very first operation attempted, a subdoc write to an ATR doc, fails with this:

          Transaction logger:14/Thread-1/aa65cd42-bd16-493b-a72d-d60c543d3e38/b448c6f6-575c-417f-8ec1-c5f38a6c8ccb error com.couchbase.client.core.error.DurabilityLevelNotAvailableException while setting ATR bucket-1/_default/atr-896-#878 to Pending

          After that, all operations timeout.

          After some digging I found MB-34108, looks like this is the same issue.  Sharath Sulochana can you please upgrade to Java client alpha 3?  Hopefully that'll fix.

          graham.pople Graham Pople added a comment - The very first operation attempted, a subdoc write to an ATR doc, fails with this: Transaction logger:14/Thread-1/aa65cd42-bd16-493b-a72d-d60c543d3e38/b448c6f6-575c-417f-8ec1-c5f38a6c8ccb error com.couchbase.client.core.error.DurabilityLevelNotAvailableException while setting ATR bucket-1/_default/atr-896-#878 to Pending After that, all operations timeout. After some digging I found MB-34108 , looks like this is the same issue.  Sharath Sulochana can you please upgrade to Java client alpha 3?  Hopefully that'll fix.

          Graham Pople Michael Nitschinger 

          YCSB build is failing to compile with new java-client 3.0.0-alpha.3 . Basically its failing to find create method in com.couchbase.client.core.service.KeyValueServiceConfig

           

          pom.xml - 

          <dependency>
          <groupId>com.couchbase.client</groupId>
          <artifactId>java-client</artifactId>
          <version>3.0.0-alpha.3</version>
          </dependency>

           

           

          maven log: 

           

          [INFO] Couchbase Java SDK 3.x Binding 0.15.0 .............. FAILURE [  0.320 s]

          [INFO] ------------------------------------------------------------------------

          [INFO] BUILD FAILURE

          [INFO] ------------------------------------------------------------------------

          [INFO] Total time: 7.462 s

          [INFO] Finished at: 2019-05-15T12:25:49-07:00

          [INFO] ------------------------------------------------------------------------

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project couchbase3-binding: Compilation failure

          [ERROR] /Users/sharathsulochana/ycsb_trnasactions/YCSB_2/YCSB/couchbase3/src/main/java/com/yahoo/ycsb/db/couchbase3/Couchbase3Client.java:[68,83] cannot find symbol

          [ERROR]   symbol:   method create(int)

          [ERROR]   location: class com.couchbase.client.core.service.KeyValueServiceConfig

          sharath.sulochana Sharath Sulochana (Inactive) added a comment - - edited Graham Pople Michael Nitschinger   YCSB build is failing to compile with new java-client 3.0.0-alpha.3 . Basically its failing to find create method in  com.couchbase.client.core.service.KeyValueServiceConfig .    pom.xml -   <dependency> <groupId>com.couchbase.client</groupId> <artifactId>java-client</artifactId> <version> 3.0.0-alpha.3 </version> </dependency>     maven log:     [INFO] Couchbase Java SDK 3.x Binding 0.15.0 .............. FAILURE [  0.320 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 7.462 s [INFO] Finished at: 2019-05-15T12:25:49-07:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project couchbase3-binding: Compilation failure [ERROR] /Users/sharathsulochana/ycsb_trnasactions/YCSB_2/YCSB/couchbase3/src/main/java/com/yahoo/ycsb/db/couchbase3/Couchbase3Client.java: [68,83] cannot find symbol [ERROR]   symbol:   method create(int) [ERROR]   location: class com.couchbase.client.core.service.KeyValueServiceConfig
          graham.pople Graham Pople added a comment -

          As Java SDK is in alpha currently and we're iterating to find the right interface, these kinds of breakages aren't unexpected.

          I took a quick look at the code and looks like you want to change to something like:

          KeyValueServiceConfig.builder().endpoints(50) 

          Can you give that a try?  You may need to iterate a little - it's a little late here for me to dig out my IDE but let me know if you get stuck and I'll send over a corrected code snippet tomorrow.

          graham.pople Graham Pople added a comment - As Java SDK is in alpha currently and we're iterating to find the right interface, these kinds of breakages aren't unexpected. I took a quick look at the code and looks like you want to change to something like: KeyValueServiceConfig.builder().endpoints( 50 ) Can you give that a try?  You may need to iterate a little - it's a little late here for me to dig out my IDE but let me know if you get stuck and I'll send over a corrected code snippet tomorrow.
          ingenthr Matt Ingenthron added a comment - - edited

          Assigning to Sharath, as next action is to make the change Graham requested. Please get with Graham and or reassign if the request doesn't unblock you.

          ingenthr Matt Ingenthron added a comment - - edited Assigning to Sharath, as next action is to make the change Graham requested. Please get with Graham and or reassign if the request doesn't unblock you.

          Changing it to builder from create worked . Thanks .

          sharath.sulochana Sharath Sulochana (Inactive) added a comment - Changing it to builder from create worked . Thanks .

          People

            sharath.sulochana Sharath Sulochana (Inactive)
            sharath.sulochana Sharath Sulochana (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty