Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50289

[Windows] [Upgrade]- Disabling n2n encryption fails with "ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded" after upgrade from 6.6.4 -> 6.6.5

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 1
    • No

    Description

      Steps to Repro
      1. Create a 4 node cluster on 6.6.4 with all the services enabled.
      2. Upgrade 6.6.4 cluster to 6.6.5 using online upgrade with swap rebalance.
      3. Enable TLS on 6.6.5 using following commands. This works fine.

      1. curl -v -u Administrator:password -X POST http://localhost:8091/internalSettings -d "canEnableStrictEncryption=true"
      2. enable n2n encryption
      /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --enable
      3. Enforce it to strict
      curl -v -u Administrator:password http://localhost:8091/settings/security -d "clusterEncryptionLevel=strict"

      4. Disable TLS

      1. Bring it back to control from strict
      curl -v -u Administrator:password http://localhost:8091/settings/security -d "clusterEncryptionLevel=control"
      2. Disable n2n encryption
      /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable

      The disable n2n command repeatedly fails as shown below.

      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable
      ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded
       
      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/bin
      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable
      ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded
       
      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/bin
      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable
      ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded
       
      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/bin
      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable
      ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded
       
      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/bin
      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable^C
       
      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/bin
      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable
      ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded
       
      Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/bin
      $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable
      ERROR: _ - Reconnect to 'ns_1@172.23.136.156' retries exceeded
      

      pools/default after setting "clusterEncryptionLevel=control" shows its set successfully. See pools_default_after_clusterEncryptionLevel_to_control.txt

      Looks similar to MB-44372. Wonder if that needs to be backported.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.0.4-7209 contains ns_server commit e5ac190 with commit message:
          MB-50289: Verify otp connectivity when opening external port

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7209 contains ns_server commit e5ac190 with commit message: MB-50289 : Verify otp connectivity when opening external port

          Build couchbase-server-7.0.4-7209 contains ns_server commit ca6926c with commit message:
          MB-50289: Modify ns_cluster:verify_otp_connection to support TLS

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7209 contains ns_server commit ca6926c with commit message: MB-50289 : Modify ns_cluster:verify_otp_connection to support TLS

          Build couchbase-server-7.0.4-7209 contains ns_server commit 6249b5b with commit message:
          MB-50289: Refactor cb_epmd:port_please code in order to ...

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7209 contains ns_server commit 6249b5b with commit message: MB-50289 : Refactor cb_epmd:port_please code in order to ...
          wayne Wayne Siu added a comment -

          Balakumaran Gopal 

          Can you also verify the fix in 7.0.4?  Thanks.

          wayne Wayne Siu added a comment - Balakumaran Gopal   Can you also verify the fix in 7.0.4?  Thanks.

          Validated after upgrade from 6.6.5-10080 -> 7.0.4-7238.

          Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin
          $ cat ../VERSION.txt 
          7.0.4-7238
           
          Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin
          $ 
           
          $ ./couchbase-cli.exe  node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable --no-ssl-verify
          WARNING: sub-command requires multi-node communication via TLS enabled ports, '--cacert' or '--no-ssl-verify' may need to be supplied
          ERROR: nodeEncryption - Can't disable nodeEncryption when the cluster encryption level has been set to strict
           
          Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin
          $ ./couchbase-cli.exe  node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable --no-ssl-verify
          Turned off encryption for node: http://172.23.136.106:8091
          Turned off encryption for node: http://172.23.136.118:8091
          Turned off encryption for node: http://172.23.136.120:8091
          Turned off encryption for node: http://172.23.136.121:8091
          SUCCESS: Switched node-to-node encryption off
           
          Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin
          

          Balakumaran.Gopal Balakumaran Gopal added a comment - Validated after upgrade from 6.6.5-10080 -> 7.0.4-7238. Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin $ cat ../VERSION.txt 7.0.4-7238   Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin $   $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable --no-ssl-verify WARNING: sub-command requires multi-node communication via TLS enabled ports, '--cacert' or '--no-ssl-verify' may need to be supplied ERROR: nodeEncryption - Can't disable nodeEncryption when the cluster encryption level has been set to strict   Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin $ ./couchbase-cli.exe node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable --no-ssl-verify Turned off encryption for node: http://172.23.136.106:8091 Turned off encryption for node: http://172.23.136.118:8091 Turned off encryption for node: http://172.23.136.120:8091 Turned off encryption for node: http://172.23.136.121:8091 SUCCESS: Switched node-to-node encryption off   Administrator@WIN-HS1RJUPNDMD /cygdrive/c/Program Files/Couchbase/Server/bin

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty