Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48901

[TLS] transaction fetch error in strict mode

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.1.0
    • 7.1.0
    • query
    • 7.1.0-1472
    • Untriaged
    • 1
    • Yes

    Description

      To repro:

      • setup single node
      • load travel-sample
      • enable n2n encryption
      • enable tls strict mode
      • run query as transaction: select * from `travel-sample` limit 1;

      [
        {
          "cause": {
            "cause": {
              "-": {
                "InnerError": {
                  "InnerError": {},
                  "Message": "unambiguous timeout"
                }
              },
              "i": "0x0",
              "s": "LookupIn",
              "t": 2500793
            },
            "raise": "failed",
            "retry": true,
            "rollback": false
          },
          "code": 17017,
          "msg": "Transaction fetch error"
        }
      ] 

       

      See attached log.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          pierre.regazzoni Pierre Regazzoni created issue -
          kamini.jagtiani Kamini Jagtiani made changes -
          Field Original Value New Value
          Assignee Kamini Jagtiani [ kamini.jagtiani ] Isha Kandaswamy [ isha ]
          isha Isha Kandaswamy made changes -
          Assignee Isha Kandaswamy [ isha ] Sitaram Vemulapalli [ sitaram.vemulapalli ]
          isha Isha Kandaswamy made changes -
          Assignee Sitaram Vemulapalli [ sitaram.vemulapalli ] Isha Kandaswamy [ isha ]

          Couldn't repro the exact error, I think it was caused by certs. So can we try again after Sitaram and my fix (both fixes need to be in the build) is picked up ? 

           

          isha Isha Kandaswamy added a comment - Couldn't repro the exact error, I think it was caused by certs. So can we try again after Sitaram and my fix (both fixes need to be in the build) is picked up ?   
          isha Isha Kandaswamy made changes -
          Assignee Isha Kandaswamy [ isha ] Pierre Regazzoni [ JIRAUSER25157 ]
          pierre.regazzoni Pierre Regazzoni made changes -
          Assignee Pierre Regazzoni [ JIRAUSER25157 ] Isha Kandaswamy [ isha ]

          With 7.1.0-1488

          cbq> begin work;
          {
              "requestID": "88cb0a77-68a7-4f80-b3dd-91e4c6f5dd0e",
              "signature": "json",
              "results": [
              ],
              "errors": [
                  {
                      "code": 5000,
                      "msg": "gcagent client initialization failed - cause: x509KeyPair: private key does not match public key"
                  }
              ],
              "status": "errors",
              "metrics": {
                  "elapsedTime": "3.025741ms",
                  "executionTime": "2.82403ms",
                  "resultCount": 0,
                  "resultSize": 0,
                  "serviceLoad": 6,
                  "errorCount": 1
              }
          }
           

          pierre.regazzoni Pierre Regazzoni added a comment - With 7.1.0-1488 cbq> begin work; { "requestID" : "88cb0a77-68a7-4f80-b3dd-91e4c6f5dd0e" , "signature" : "json" , "results" : [ ], "errors" : [ { "code" : 5000 , "msg" : "gcagent client initialization failed - cause: x509KeyPair: private key does not match public key" } ], "status" : "errors" , "metrics" : { "elapsedTime" : "3.025741ms" , "executionTime" : "2.82403ms" , "resultCount" : 0 , "resultSize" : 0 , "serviceLoad" : 6 , "errorCount" : 1 } }
          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

          The above error one we fixed, wait for next build.
          The fix might have fix the original reported problem. It might have happened in following scenario.
          completed transaction.
          n2n enabled
          now again start transaction.
          when n2n enabled it tries to change root certificate but due to wrongly processing it is not loaded, connection might be using nil certificate/old one causing timeout.

          Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited The above error one we fixed, wait for next build. The fix might have fix the original reported problem. It might have happened in following scenario. completed transaction. n2n enabled now again start transaction. when n2n enabled it tries to change root certificate but due to wrongly processing it is not loaded, connection might be using nil certificate/old one causing timeout.

          Build couchbase-server-7.1.0-1489 contains query commit fd3d2a1 with commit message:
          MB-48901: Dont overwrite certfile prior to setting values

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1489 contains query commit fd3d2a1 with commit message: MB-48901 : Dont overwrite certfile prior to setting values

          Build couchbase-server-7.1.0-1489 contains query commit c1beada with commit message:
          MB-48901. don't do pcks8 on root certificate

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1489 contains query commit c1beada with commit message: MB-48901 . don't do pcks8 on root certificate
          pierre.regazzoni Pierre Regazzoni made changes -
          Assignee Isha Kandaswamy [ isha ] Pierre Regazzoni [ JIRAUSER25157 ]

          Verified on 7.1.0-1489 ... working now:

          cbq> begin work;
          {
              "requestID": "1f5fc8e9-517c-4532-93c9-3583f923321c",
              "signature": "json",
              "results": [
              {
                  "txid": "47718c24-412a-4231-b599-d8a0be7d26e4"
              }
              ],
              "status": "success",
              "metrics": {
                  "elapsedTime": "2.544247ms",
                  "executionTime": "2.353106ms",
                  "resultCount": 1,
                  "resultSize": 62,
                  "serviceLoad": 6,
                  "transactionElapsedTime": "61.945µs",
                  "transactionRemainingTime": "1m59.999919975s"
              }
          }
          cbq> select * from `travel-sample` limit 1;
          {
              "requestID": "9389cfe8-87e6-4cc9-a935-28974a1d02a1",
              "signature": {
                  "*": "*"
              },
              "results": [
              {
                  "travel-sample": {
                      "callsign": "MILE-AIR",
                      "country": "United States",
                      "iata": "Q5",
                      "icao": "MLA",
                      "id": 10,
                      "name": "40-Mile Air",
                      "type": "airline"
                  }
              }
              ],
              "status": "success",
              "metrics": {
                  "elapsedTime": "107.320521ms",
                  "executionTime": "107.009682ms",
                  "resultCount": 1,
                  "resultSize": 260,
                  "serviceLoad": 1,
                  "transactionElapsedTime": "2.636856543s",
                  "transactionRemainingTime": "1m57.363102861s"
              }
          }
          cbq> commit;
          {
              "requestID": "e288f95e-594c-4ee0-9c8d-a4f25ab34e8b",
              "signature": "json",
              "results": [
              ],
              "status": "success",
              "metrics": {
                  "elapsedTime": "800.952µs",
                  "executionTime": "632.72µs",
                  "resultCount": 0,
                  "resultSize": 0,
                  "serviceLoad": 1,
                  "transactionElapsedTime": "9.98609078s"
              }
          }
           

          pierre.regazzoni Pierre Regazzoni added a comment - Verified on 7.1.0-1489 ... working now: cbq> begin work; { "requestID" : "1f5fc8e9-517c-4532-93c9-3583f923321c" , "signature" : "json" , "results" : [ { "txid" : "47718c24-412a-4231-b599-d8a0be7d26e4" } ], "status" : "success" , "metrics" : { "elapsedTime" : "2.544247ms" , "executionTime" : "2.353106ms" , "resultCount" : 1 , "resultSize" : 62 , "serviceLoad" : 6 , "transactionElapsedTime" : "61.945µs" , "transactionRemainingTime" : "1m59.999919975s" } } cbq> select * from `travel-sample` limit 1 ; { "requestID" : "9389cfe8-87e6-4cc9-a935-28974a1d02a1" , "signature" : { "*" : "*" }, "results" : [ { "travel-sample" : { "callsign" : "MILE-AIR" , "country" : "United States" , "iata" : "Q5" , "icao" : "MLA" , "id" : 10 , "name" : "40-Mile Air" , "type" : "airline" } } ], "status" : "success" , "metrics" : { "elapsedTime" : "107.320521ms" , "executionTime" : "107.009682ms" , "resultCount" : 1 , "resultSize" : 260 , "serviceLoad" : 1 , "transactionElapsedTime" : "2.636856543s" , "transactionRemainingTime" : "1m57.363102861s" } } cbq> commit; { "requestID" : "e288f95e-594c-4ee0-9c8d-a4f25ab34e8b" , "signature" : "json" , "results" : [ ], "status" : "success" , "metrics" : { "elapsedTime" : "800.952µs" , "executionTime" : "632.72µs" , "resultCount" : 0 , "resultSize" : 0 , "serviceLoad" : 1 , "transactionElapsedTime" : "9.98609078s" } }
          pierre.regazzoni Pierre Regazzoni made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          pierre.regazzoni Pierre Regazzoni made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          isha Isha Kandaswamy made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]

          Looks like when I install pkcs certs I see a similar issue with xls. Im reopening this bug to debug some more.

          isha Isha Kandaswamy added a comment - Looks like when I install pkcs certs I see a similar issue with xls. Im reopening this bug to debug some more.

          Build couchbase-server-7.1.0-1498 contains query commit 9a582ef with commit message:
          MB-48901. Add logging of certificates refresh

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1498 contains query commit 9a582ef with commit message: MB-48901 . Add logging of certificates refresh
          Sitaram.Vemulapalli Sitaram Vemulapalli made changes -
          Assignee Pierre Regazzoni [ JIRAUSER25157 ] Isha Kandaswamy [ isha ]

          Pierres suggestion worked for me, 

          cat > client-auth-settings.json <<EOF

          {"prefixes": [\{"delimiter": "", "path": "subject.cn", "prefix": ""}

          ], "state": "enable"}
          EOF
          /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --set-client-auth ./client-auth-settings.json
          /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --client-auth

          isha Isha Kandaswamy added a comment - Pierres suggestion worked for me,  cat > client-auth-settings.json <<EOF {"prefixes": [\{"delimiter": "", "path": "subject.cn", "prefix": ""} ], "state": "enable"} EOF /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --set-client-auth ./client-auth-settings.json /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost -u Administrator -p password --client-auth
          isha Isha Kandaswamy made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          pierre.regazzoni Pierre Regazzoni made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            isha Isha Kandaswamy
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty