Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48901

[TLS] transaction fetch error in strict mode

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.1.0
    • 7.1.0
    • query
    • 7.1.0-1472
    • Untriaged
    • 1
    • Yes

    Description

      To repro:

      • setup single node
      • load travel-sample
      • enable n2n encryption
      • enable tls strict mode
      • run query as transaction: select * from `travel-sample` limit 1;

      [
        {
          "cause": {
            "cause": {
              "-": {
                "InnerError": {
                  "InnerError": {},
                  "Message": "unambiguous timeout"
                }
              },
              "i": "0x0",
              "s": "LookupIn",
              "t": 2500793
            },
            "raise": "failed",
            "retry": true,
            "rollback": false
          },
          "code": 17017,
          "msg": "Transaction fetch error"
        }
      ] 

       

      See attached log.

      Attachments

        For Gerrit Dashboard: MB-48901
        # Subject Branch Project Status CR V

        Activity

          Couldn't repro the exact error, I think it was caused by certs. So can we try again after Sitaram and my fix (both fixes need to be in the build) is picked up ? 

           

          isha Isha Kandaswamy added a comment - Couldn't repro the exact error, I think it was caused by certs. So can we try again after Sitaram and my fix (both fixes need to be in the build) is picked up ?   

          With 7.1.0-1488

          cbq> begin work;
          {
              "requestID": "88cb0a77-68a7-4f80-b3dd-91e4c6f5dd0e",
              "signature": "json",
              "results": [
              ],
              "errors": [
                  {
                      "code": 5000,
                      "msg": "gcagent client initialization failed - cause: x509KeyPair: private key does not match public key"
                  }
              ],
              "status": "errors",
              "metrics": {
                  "elapsedTime": "3.025741ms",
                  "executionTime": "2.82403ms",
                  "resultCount": 0,
                  "resultSize": 0,
                  "serviceLoad": 6,
                  "errorCount": 1
              }
          }
           

          pierre.regazzoni Pierre Regazzoni added a comment - With 7.1.0-1488 cbq> begin work; { "requestID" : "88cb0a77-68a7-4f80-b3dd-91e4c6f5dd0e" , "signature" : "json" , "results" : [ ], "errors" : [ { "code" : 5000 , "msg" : "gcagent client initialization failed - cause: x509KeyPair: private key does not match public key" } ], "status" : "errors" , "metrics" : { "elapsedTime" : "3.025741ms" , "executionTime" : "2.82403ms" , "resultCount" : 0 , "resultSize" : 0 , "serviceLoad" : 6 , "errorCount" : 1 } }
          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

          The above error one we fixed, wait for next build.
          The fix might have fix the original reported problem. It might have happened in following scenario.
          completed transaction.
          n2n enabled
          now again start transaction.
          when n2n enabled it tries to change root certificate but due to wrongly processing it is not loaded, connection might be using nil certificate/old one causing timeout.

          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited The above error one we fixed, wait for next build. The fix might have fix the original reported problem. It might have happened in following scenario. completed transaction. n2n enabled now again start transaction. when n2n enabled it tries to change root certificate but due to wrongly processing it is not loaded, connection might be using nil certificate/old one causing timeout.

          Build couchbase-server-7.1.0-1489 contains query commit fd3d2a1 with commit message:
          MB-48901: Dont overwrite certfile prior to setting values

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1489 contains query commit fd3d2a1 with commit message: MB-48901 : Dont overwrite certfile prior to setting values

          Build couchbase-server-7.1.0-1489 contains query commit c1beada with commit message:
          MB-48901. don't do pcks8 on root certificate

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1489 contains query commit c1beada with commit message: MB-48901 . don't do pcks8 on root certificate

          Verified on 7.1.0-1489 ... working now:

          cbq> begin work;
          {
              "requestID": "1f5fc8e9-517c-4532-93c9-3583f923321c",
              "signature": "json",
              "results": [
              {
                  "txid": "47718c24-412a-4231-b599-d8a0be7d26e4"
              }
              ],
              "status": "success",
              "metrics": {
                  "elapsedTime": "2.544247ms",
                  "executionTime": "2.353106ms",
                  "resultCount": 1,
                  "resultSize": 62,
                  "serviceLoad": 6,
                  "transactionElapsedTime": "61.945µs",
                  "transactionRemainingTime": "1m59.999919975s"
              }
          }
          cbq> select * from `travel-sample` limit 1;
          {
              "requestID": "9389cfe8-87e6-4cc9-a935-28974a1d02a1",
              "signature": {
                  "*": "*"
              },
              "results": [
              {
                  "travel-sample": {
                      "callsign": "MILE-AIR",
                      "country": "United States",
                      "iata": "Q5",
                      "icao": "MLA",
                      "id": 10,
                      "name": "40-Mile Air",
                      "type": "airline"
                  }
              }
              ],
              "status": "success",
              "metrics": {
                  "elapsedTime": "107.320521ms",
                  "executionTime": "107.009682ms",
                  "resultCount": 1,
                  "resultSize": 260,
                  "serviceLoad": 1,
                  "transactionElapsedTime": "2.636856543s",
                  "transactionRemainingTime": "1m57.363102861s"
              }
          }
          cbq> commit;
          {
              "requestID": "e288f95e-594c-4ee0-9c8d-a4f25ab34e8b",
              "signature": "json",
              "results": [
              ],
              "status": "success",
              "metrics": {
                  "elapsedTime": "800.952µs",
                  "executionTime": "632.72µs",
                  "resultCount": 0,
                  "resultSize": 0,
                  "serviceLoad": 1,
                  "transactionElapsedTime": "9.98609078s"
              }
          }
           

          pierre.regazzoni Pierre Regazzoni added a comment - Verified on 7.1.0-1489 ... working now: cbq> begin work; { "requestID" : "1f5fc8e9-517c-4532-93c9-3583f923321c" , "signature" : "json" , "results" : [ { "txid" : "47718c24-412a-4231-b599-d8a0be7d26e4" } ], "status" : "success" , "metrics" : { "elapsedTime" : "2.544247ms" , "executionTime" : "2.353106ms" , "resultCount" : 1 , "resultSize" : 62 , "serviceLoad" : 6 , "transactionElapsedTime" : "61.945µs" , "transactionRemainingTime" : "1m59.999919975s" } } cbq> select * from `travel-sample` limit 1 ; { "requestID" : "9389cfe8-87e6-4cc9-a935-28974a1d02a1" , "signature" : { "*" : "*" }, "results" : [ { "travel-sample" : { "callsign" : "MILE-AIR" , "country" : "United States" , "iata" : "Q5" , "icao" : "MLA" , "id" : 10 , "name" : "40-Mile Air" , "type" : "airline" } } ], "status" : "success" , "metrics" : { "elapsedTime" : "107.320521ms" , "executionTime" : "107.009682ms" , "resultCount" : 1 , "resultSize" : 260 , "serviceLoad" : 1 , "transactionElapsedTime" : "2.636856543s" , "transactionRemainingTime" : "1m57.363102861s" } } cbq> commit; { "requestID" : "e288f95e-594c-4ee0-9c8d-a4f25ab34e8b" , "signature" : "json" , "results" : [ ], "status" : "success" , "metrics" : { "elapsedTime" : "800.952µs" , "executionTime" : "632.72µs" , "resultCount" : 0 , "resultSize" : 0 , "serviceLoad" : 1 , "transactionElapsedTime" : "9.98609078s" } }

          Looks like when I install pkcs certs I see a similar issue with xls. Im reopening this bug to debug some more.

          isha Isha Kandaswamy added a comment - Looks like when I install pkcs certs I see a similar issue with xls. Im reopening this bug to debug some more.

          Build couchbase-server-7.1.0-1498 contains query commit 9a582ef with commit message:
          MB-48901. Add logging of certificates refresh

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1498 contains query commit 9a582ef with commit message: MB-48901 . Add logging of certificates refresh

          Pierres suggestion worked for me, 

          cat > client-auth-settings.json <<EOF

          {"prefixes": [\{"delimiter": "", "path": "subject.cn", "prefix": ""}

          ], "state": "enable"}
          EOF
          /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --set-client-auth ./client-auth-settings.json
          /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --client-auth

          isha Isha Kandaswamy added a comment - Pierres suggestion worked for me,  cat > client-auth-settings.json <<EOF {"prefixes": [\{"delimiter": "", "path": "subject.cn", "prefix": ""} ], "state": "enable"} EOF /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --set-client-auth ./client-auth-settings.json /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --client-auth

          People

            isha Isha Kandaswamy
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty