Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-602

SG is failed to connect to CBS with x509 cert auth

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.7.0
    • 2.7.0
    • SyncGateway
    • Security Level: Public
    • None
    • Hide
      Mandatory:
       - CBL / SG Version:
         - SG Config:
       - Steps to Reproduce:
       - Actual Result:
       - Expected Result:
       - Logs :
            SGW LOGS: sgcollect info
            CBL LOGS:
            Logcat LOGS: for Android tickets
       - Github link for the code:
       - Jenkins job failure link:
       - Pytest Command
       - What is the last build this test passed:
      Show
      Mandatory:  - CBL / SG Version:    - SG Config:  - Steps to Reproduce:  - Actual Result:  - Expected Result:  - Logs :       SGW LOGS: sgcollect info       CBL LOGS:       Logcat LOGS: for Android tickets  - Github link for the code:  - Jenkins job failure link:  - Pytest Command  - What is the last build this test passed:
    • CBG Sprint 35, CBG Sprint 36
    • 3
    • Critical

    Description

      SG 2.7.0 is not able to connect to CBS with x509 cert auth. Connection is working fine with password base auth.

       

      Environment: 

      SG 2.7.0-137

      CBS 6.5.0-4821

      CBL (iOS) - 2.7.0-91

       

      Steps to Reproduce:

      1. Install CBS and use gen_keystore.sh to install and generate certs for x509 based auth.Example command: ./gen_keystore.sh 172.16.1.205 travel-sample
      2. Install SG and use below sync_gateway.json 

        {
                "adminInterface": "0.0.0.0:4985", 
                "compressResponses": false, 
                "databases": {
                    "db": {
                        "allow_conflicts": false, 
                        "bucket": "travel-sample", 
                        "cacertpath": "/home/sync_gateway/certs/ca.pem", 
                        "certpath": "/home/sync_gateway/certs/chain.pem", 
                        "enable_shared_bucket_access": true, 
                        "import_docs": "continuous", 
                        "keypath": "/home/sync_gateway/certs/pkey.key", 
                        "server": "couchbases://172.16.1.205:", 
                        "use_views": true
                    }
                }, 
                "interface": ":4984", 
                "logging": {
                    "debug": {
                        "enabled": true
                    }
                }, 
                "maxFileDescriptors": 90000, 
                "maxIncomingConnections": 0
            }
        }

      3. Run command service sync_gateway start

      Actual Result:
      SG fails to start and throws below error -

      2019-11-20T12:40:27.746+05:30 ==== Couchbase Sync Gateway/2.7.0(137;05d5126) EE ====
      2019-11-20T12:40:27.746+05:30 ==== Couchbase Sync Gateway/2.7.0(137;05d5126) EE ====
      2019-11-20T12:40:27.865+05:30 [ERR] cbgt index creation failed: manager_api: failed to connect to or retrieve information from source, sourceType: couchbase, sourceName: travel-sample, sourceUUID: e56f6f4890b825fcabafdf1c1c2d0f4c, err: gocouchbase_helper: CouchbaseBucket connection failed, server: https://172.16.1.205:18091, poolName: default, bucketName: travel-sample, sourceParams: "{\"includeXAttrs\":true}", err: HTTP error 401 Unauthorized getting "https://172.16.1.205:18091/pools": , please check that your authUser and authPassword are correct and that your couchbase cluster ("https://172.16.1.205:18091") is available -- base.(*CbgtContext).StartManager() at dcp_sharded.go:286
      2019-11-20T12:40:27.865+05:30 [ERR] Error opening database db: manager_api: failed to connect to or retrieve information from source, sourceType: couchbase, sourceName: travel-sample, sourceUUID: e56f6f4890b825fcabafdf1c1c2d0f4c, err: gocouchbase_helper: CouchbaseBucket connection failed, server: https://172.16.1.205:18091, poolName: default, bucketName: travel-sample, sourceParams: "{\"includeXAttrs\":true}", err: HTTP error 401 Unauthorized getting "https://172.16.1.205:18091/pools": , please check that your authUser and authPassword are correct and that your couchbase cluster ("https://172.16.1.205:18091") is available -- rest.RunServer() at config.go:1020

      Expected Result:

      SG should be able to connect to CBS with x509 cert without any error

      Logs :

      Available in attachment fields

       

      Steps to generate Certs and installed in CBS:

      1. Download all the files available here - https://github.com/couchbaselabs/mobile-testkit/tree/master/resources/x509_cert_gen

      2. Add CBS IP at the last line of openssl-san.cnf

      (venv) LFC:certs lfc$ cat openssl-san.cnf 
      [req]
      x509_extensions = v3_req
      distinguished_name = req_distinguished_name
       
       
      [req_distinguished_name]
       
       
      [ v3_req ]
       
       
      # Extensions to add to a certificate request
       
       
      basicConstraints = CA:true
      keyUsage = nonRepudiation, digitalSignature, keyEncipherment
      subjectAltName = @alt_names
       
       
      [alt_names]
      IP.1 = 172.16.1.205

      3. Create a user "travel-sample" in CBS with "Full Admin" access

      4. run command  ./gen_keystore.sh <CBS_IP> <CBS_USERNAME> <CBS_HOST_SSH_PASSWORD>

       ./gen_keystore.sh 172.16.1.205 travel-sample couchbase

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Adam Fraser and Sridevi Saragadam,

            So here are my findings - 

            Tests are passing with CBS 6.0.3 with and without x509. However, with 6.5.0 tests are passing only when indexing is through views but for GSI, SG is not able to connect to CBS.

             

            Logs for SG and CBS 6.5.0 cluster - CBS_6.5.0-2783_SSL_2.7.0_145.zip

             

             

            hemant.rajput Hemant Rajput added a comment - Adam Fraser  and Sridevi Saragadam , So here are my findings -  Tests are passing with CBS 6.0.3 with and without x509. However, with 6.5.0 tests are passing only when indexing is through views but for GSI, SG is not able to connect to CBS.   Logs for SG and CBS 6.5.0 cluster -  CBS_6.5.0-2783_SSL_2.7.0_145.zip    
            ben.brooks Ben Brooks added a comment -

            6.5.0-4912 seems to contain the fixes for MB-37083. I'll give it a quick sanity test.

            ben.brooks Ben Brooks added a comment - 6.5.0-4912 seems to contain the fixes for MB-37083 . I'll give it a quick sanity test.
            ben.brooks Ben Brooks added a comment -

            Verified SG connects to query service out of the box using SSL using 6.5.0-4912

            12:06 $ go build && ./sync_gateway sg_config.json
            2019-12-03T12:06:32.678Z ==== Couchbase Sync Gateway/master(964b42e) CE ====
            2019-12-03T12:06:32.678Z [INF] Logging: Console to stderr
            2019-12-03T12:06:32.678Z [INF] Logging: Files disabled
            2019-12-03T12:06:32.678Z [INF] Logging: Console level: debug
            2019-12-03T12:06:32.678Z [INF] Logging: Console keys: [HTTP Query]
            2019-12-03T12:06:32.678Z [INF] Logging: Redaction level: none
            2019-12-03T12:06:32.678Z [INF] Configured process to allow 5000 open file descriptors
            2019-12-03T12:06:32.678Z [INF] Logging stats with frequency: 1m0s
            2019-12-03T12:06:32.678Z [INF] Opening db /db1 as bucket "default", pool "default", server <couchbases://10.112.195.101>
            2019-12-03T12:06:32.678Z [INF] GoCBCustomSGTranscoder Opening Couchbase database default on <couchbases://10.112.195.101> as user "Administrator"
            2019-12-03T12:06:32.684Z [INF] Successfully opened bucket default
            2019-12-03T12:06:32.699Z [INF] Set query timeouts for bucket default to cluster:1m15s, bucket:1m15s
            2019-12-03T12:06:32.699Z [INF] Initializing indexes with numReplicas: 0...
            2019-12-03T12:06:32.750Z [INF] Verifying index availability for bucket default...
            2019-12-03T12:06:32.750Z [DBG] Query+: Verifying index availability for index sg_channels_x1...
            2019-12-03T12:06:32.750Z [DBG] Query+: Verifying index availability for index sg_access_x1...
            2019-12-03T12:06:32.750Z [DBG] Query+: Verifying index availability for index sg_roleAccess_x1...
            2019-12-03T12:06:32.756Z [DBG] Query+: Index sg_roleAccess_x1 verified as ready
            2019-12-03T12:06:32.756Z [DBG] Query+: Index sg_access_x1 verified as ready
            2019-12-03T12:06:32.757Z [DBG] Query+: Index sg_channels_x1 verified as ready
            2019-12-03T12:06:32.757Z [INF] Indexes ready for bucket default.
            2019-12-03T12:06:32.757Z [INF] delta_sync enabled=false with rev_max_age_seconds=86400 for database db1
            2019-12-03T12:06:32.757Z [INF] Created background task: "CleanAgedItems" with interval 1m0s
            2019-12-03T12:06:32.757Z [INF] Created background task: "InsertPendingEntries" with interval 2.5s
            2019-12-03T12:06:32.757Z [INF] Created background task: "CleanSkippedSequenceQueue" with interval 30m0s
            2019-12-03T12:06:32.785Z [INF] Using metadata purge interval of 3.00 days for tombstone compaction.
            2019-12-03T12:06:32.786Z [WRN] Automatic compaction can only be enabled on nodes running an Import process -- db.NewDatabaseContext() at database.go:383
            2019-12-03T12:06:32.789Z [INF] Using default sync function 'channel(doc.channels)' for database "db1"
            2019-12-03T12:06:32.789Z [INF] Starting admin server on 127.0.0.1:4985
            2019-12-03T12:06:32.790Z [INF] Starting server on :4984 ...
            

            ben.brooks Ben Brooks added a comment - Verified SG connects to query service out of the box using SSL using 6.5.0-4912 12:06 $ go build && ./sync_gateway sg_config.json 2019-12-03T12:06:32.678Z ==== Couchbase Sync Gateway/master(964b42e) CE ==== 2019-12-03T12:06:32.678Z [INF] Logging: Console to stderr 2019-12-03T12:06:32.678Z [INF] Logging: Files disabled 2019-12-03T12:06:32.678Z [INF] Logging: Console level: debug 2019-12-03T12:06:32.678Z [INF] Logging: Console keys: [HTTP Query] 2019-12-03T12:06:32.678Z [INF] Logging: Redaction level: none 2019-12-03T12:06:32.678Z [INF] Configured process to allow 5000 open file descriptors 2019-12-03T12:06:32.678Z [INF] Logging stats with frequency: 1m0s 2019-12-03T12:06:32.678Z [INF] Opening db /db1 as bucket "default", pool "default", server <couchbases://10.112.195.101> 2019-12-03T12:06:32.678Z [INF] GoCBCustomSGTranscoder Opening Couchbase database default on <couchbases://10.112.195.101> as user "Administrator" 2019-12-03T12:06:32.684Z [INF] Successfully opened bucket default 2019-12-03T12:06:32.699Z [INF] Set query timeouts for bucket default to cluster:1m15s, bucket:1m15s 2019-12-03T12:06:32.699Z [INF] Initializing indexes with numReplicas: 0... 2019-12-03T12:06:32.750Z [INF] Verifying index availability for bucket default... 2019-12-03T12:06:32.750Z [DBG] Query+: Verifying index availability for index sg_channels_x1... 2019-12-03T12:06:32.750Z [DBG] Query+: Verifying index availability for index sg_access_x1... 2019-12-03T12:06:32.750Z [DBG] Query+: Verifying index availability for index sg_roleAccess_x1... 2019-12-03T12:06:32.756Z [DBG] Query+: Index sg_roleAccess_x1 verified as ready 2019-12-03T12:06:32.756Z [DBG] Query+: Index sg_access_x1 verified as ready 2019-12-03T12:06:32.757Z [DBG] Query+: Index sg_channels_x1 verified as ready 2019-12-03T12:06:32.757Z [INF] Indexes ready for bucket default. 2019-12-03T12:06:32.757Z [INF] delta_sync enabled=false with rev_max_age_seconds=86400 for database db1 2019-12-03T12:06:32.757Z [INF] Created background task: "CleanAgedItems" with interval 1m0s 2019-12-03T12:06:32.757Z [INF] Created background task: "InsertPendingEntries" with interval 2.5s 2019-12-03T12:06:32.757Z [INF] Created background task: "CleanSkippedSequenceQueue" with interval 30m0s 2019-12-03T12:06:32.785Z [INF] Using metadata purge interval of 3.00 days for tombstone compaction. 2019-12-03T12:06:32.786Z [WRN] Automatic compaction can only be enabled on nodes running an Import process -- db.NewDatabaseContext() at database.go:383 2019-12-03T12:06:32.789Z [INF] Using default sync function 'channel(doc.channels)' for database "db1" 2019-12-03T12:06:32.789Z [INF] Starting admin server on 127.0.0.1:4985 2019-12-03T12:06:32.790Z [INF] Starting server on :4984 ...
            ben.brooks Ben Brooks added a comment -

            Sridevi Saragadam is this ready to close based on verification done in MB-36900 ?

            ben.brooks Ben Brooks added a comment - Sridevi Saragadam is this ready to close based on verification done in MB-36900 ?

            fixed and verified with CBS- 6.7.0-4912 and worked fine.

            sridevi.saragadam Sridevi Saragadam (Inactive) added a comment - fixed and verified with CBS- 6.7.0-4912 and worked fine.

            People

              sridevi.saragadam Sridevi Saragadam (Inactive)
              hemant.rajput Hemant Rajput
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty