Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48112

[CLI] cbtransfer based tools fail to connect to cluster when TLS is enforced

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.0.2
    • 7.1.0
    • tools
    • Untriaged
    • 1
    • No
    • Tools 2021 Dec, Tools 2022-Jan

    Description

      Build : 7.0.2-6558

      Steps to reproduce :
      1. 2 node cluster with kv+index+query services on both nodes
      2. Enforce TLS

      • disable auto-failover
      • enable N2N encryption: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --enable
      • set TLS strict mode: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost:8091 -u Administrator -p password --set --cluster-encryption-level strict
        3. Create a default bucket
        4. Try to load some docs to the cluster using cbworkloadgen.

      I tried the following commands, all errored out.

      [root@s44010 bin]# ./cbworkloadgen -u Administrator -p password -j -i 1000  --prefix=test1 --ssl -n 127.0.0.1:18091
      Unable to connect to host at http://127.0.0.1:18091: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
      

      [root@s44010 bin]# ./cbworkloadgen -u Administrator -p password -j -i 1000  --prefix=test1 --ssl -n 172.23.104.106:18091
      Unable to connect to host at http://172.23.104.106:18091: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
      

      [root@s44010 bin]# ./cbworkloadgen -u Administrator -p password -j -i 1000 --prefix=test1 --ssl
      Exception in thread s0:
      Traceback (most recent call last):
        File "/opt/couchbase/lib/python/runtime/lib/python3.9/threading.py", line 954, in _bootstrap_inner
          self.run()
        File "/opt/couchbase/lib/python/runtime/lib/python3.9/threading.py", line 892, in run
          self._target(*self._args, **self._kwargs)
        File "/opt/couchbase/lib/python/pump_mc.py", line 113, in run
          rv, batch, need_backoff = self.scatter_gather(mconns, batch)
        File "/opt/couchbase/lib/python/pump_cb.py", line 86, in scatter_gather
          rv, conn = self.find_conn(mconns, vbucket_id, msgs)
        File "/opt/couchbase/lib/python/pump_cb.py", line 474, in find_conn
          rv, conn = CBSink.connect_mc(host, port, username, password, bucket, self.opts.ssl,
        File "/opt/couchbase/lib/python/pump_mc.py", line 571, in connect_mc
          return pump.get_mcd_conn(host, port, username, password, bucket, use_ssl=use_ssl, verify=verify,
        File "/opt/couchbase/lib/python/pump.py", line 1181, in get_mcd_conn
          conn = cb_bin_client.MemcachedClient(host, port, use_ssl=use_ssl, verify=verify, cacert=ca_cert)
        File "/opt/couchbase/lib/python/cb_bin_client.py", line 118, in __init__
          raise sock_error
      UnboundLocalError: local variable 'sock_error' referenced before assignment
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-48112
          # Subject Branch Project Status CR V

          Activity

            james.lee James Lee added a comment -

            I think there's a couple separate (but related) issues which are stopping users from being able to use 'cbworkloadgen' with TLS enabled. So far I've found:
            1) It currently always assumes a non-TLS connection is created, by enforcing the prefix 'http://' to the '-n' or '--node' flag (this bypasses some validation for 'ClusterManger')
            2) When forcing TLS using the '--ssl' flag, it ignores the given port and forces the the default (11207, which doesn't cover all cases)

            I'm still investigating because I currently don't believe that these are the only issues. Looking at the code which added support for TLS to transfer related commands, I don't believe it's ever worked (or at least I couldn't get it to work when I check out to that change locally).

            james.lee James Lee added a comment - I think there's a couple separate (but related) issues which are stopping users from being able to use ' cbworkloadgen ' with TLS enabled. So far I've found: 1) It currently always assumes a non-TLS connection is created, by enforcing the prefix ' http:// ' to the ' -n ' or ' --node ' flag (this bypasses some validation for ' ClusterManger ') 2) When forcing TLS using the ' --ssl ' flag, it ignores the given port and forces the the default (11207, which doesn't cover all cases) I'm still investigating because I currently don't believe that these are the only issues. Looking at the code which added support for TLS to transfer related commands, I don't believe it's ever worked (or at least I couldn't get it to work when I check out to that change locally).

            Build couchbase-server-7.1.0-2028 contains couchbase-cli commit befd790 with commit message:
            MB-48112 Flip opts.no_ssl_verify variable logic

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2028 contains couchbase-cli commit befd790 with commit message: MB-48112 Flip opts.no_ssl_verify variable logic

            Build couchbase-server-7.1.0-2028 contains couchbase-cli commit 5788c58 with commit message:
            MB-48112 Fix socket error not being re-raised properly

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2028 contains couchbase-cli commit 5788c58 with commit message: MB-48112 Fix socket error not being re-raised properly

            Build couchbase-server-7.1.0-2048 contains couchbase-cli commit 6b29aa0 with commit message:
            MB-48112 Add SSL and cacert flags to cbworkloadgen

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2048 contains couchbase-cli commit 6b29aa0 with commit message: MB-48112 Add SSL and cacert flags to cbworkloadgen

            Build couchbase-server-7.1.0-2056 contains couchbase-cli commit 7e699b3 with commit message:
            MB-48112 Fix REST port validation breaking integration testing

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2056 contains couchbase-cli commit 7e699b3 with commit message: MB-48112 Fix REST port validation breaking integration testing
            thuan Thuan Nguyen added a comment -

            Verified on build 7.1.0-2378

            [root@s44015 ~]# /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --enable
            Turned on encryption for node: http://172.23.121.224:8091
            Turned on encryption for node: http://172.23.121.225:8091
            SUCCESS: Switched node-to-node encryption on
            [root@s44015 ~]# /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost:8091 -u Administrator -p password --set --cluster-encryption-level strict
            SUCCESS: Security settings updated
            [root@s44015 ~]#/opt/couchbase/bin//cbworkloadgen -u Administrator -p password -j -i 1000  --prefix=test1 --ssl --no-ssl-verify -n 172.23.121.224:18091
              [####################] 100.0% (1053/estimated 1053 msgs)
            bucket: default, msgs transferred...
                   :                total |       last |    per sec
             byte  :                71160 |      71160 |   194998.8
            done
            [root@s44015 ~]# 
             
            [root@s44015 ~]# /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost:8091 -u Administrator -p password --regenerate-cert /root/cacert.pem
            SUCCESS: Certificate regenerate and copied to `/root/cacert.pem`
            [root@s44015 ~]# /opt/couchbase/bin//cbworkloadgen -u Administrator -p password -j -i 1000  --prefix=test2 --ssl --cacert /root/cacert.pem  -n 172.23.121.224:18091
              [####################] 100.0% (1053/estimated 1053 msgs)
            bucket: default, msgs transferred...
                   :                total |       last |    per sec
             byte  :                71160 |      71160 |   193711.9
            done
            [root@s44015 ~]# 
             
            
            

            thuan Thuan Nguyen added a comment - Verified on build 7.1.0-2378 [root@s44015 ~]# /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --enable Turned on encryption for node: http://172.23.121.224:8091 Turned on encryption for node: http://172.23.121.225:8091 SUCCESS: Switched node-to-node encryption on [root@s44015 ~]# /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost:8091 -u Administrator -p password --set --cluster-encryption-level strict SUCCESS: Security settings updated [root@s44015 ~]#/opt/couchbase/bin//cbworkloadgen -u Administrator -p password -j -i 1000 --prefix=test1 --ssl --no-ssl-verify -n 172.23.121.224:18091 [####################] 100.0% (1053/estimated 1053 msgs) bucket: default, msgs transferred... : total | last | per sec byte : 71160 | 71160 | 194998.8 done [root@s44015 ~]#   [root@s44015 ~]# /opt/couchbase/bin/couchbase-cli ssl-manage -c localhost:8091 -u Administrator -p password --regenerate-cert /root/cacert.pem SUCCESS: Certificate regenerate and copied to `/root/cacert.pem` [root@s44015 ~]# /opt/couchbase/bin//cbworkloadgen -u Administrator -p password -j -i 1000 --prefix=test2 --ssl --cacert /root/cacert.pem -n 172.23.121.224:18091 [####################] 100.0% (1053/estimated 1053 msgs) bucket: default, msgs transferred... : total | last | per sec byte : 71160 | 71160 | 193711.9 done [root@s44015 ~]#  

            People

              thuan Thuan Nguyen
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty