Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51464

Cannot online upgrade from 6.5.x to Neo when n2n encryption w/ system generated certs

    XMLWordPrintable

Details

    • 1

    Description

      When online upgrading a cluster to Neo w/ n2n encryption enabled, the Neo node is always selected as the rebalance coordinator. For reasons unknown, when the cluster is encrypted w/ canned / system-generated certificates, the Neo node cbas process is unable to handshake w/ the 6.5.2 CC web server to monitor the rebalance. It is not known at this time if this is a regression in Neo, filed as a Neo task to determine next steps.

      All requests fail with:

      2022-03-16T01:45:22.174+00:00 INFO CBAS.cbas failing rebalance due to wait for condition on https://cbas1.couchbase.host:9111/analytics/cluster timed out

      The CC seems displeased with the certificate presented in the requests:

      2022-03-16T02:04:42.734+00:00 WARN CBAS.server.HttpServerHandler [nioEventLoopGroup-3-2] Failure handling HTTP Request
      		 
      io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
      		 
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at java.lang.Thread.run(Unknown Source) [?:?]
      		 
      Caused by: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
      		 
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1007) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1271) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1225) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1296) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1339) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) ~[netty-all-4.1.48.Final.jar:4.1.48.Final]
      		 
      	... 17 more

      Attachments

        For Gerrit Dashboard: MB-51464
        # Subject Branch Project Status CR V
        172344,2 MB-51419: disable upgradeEncryptedClusterOOTBCerts for 6.5.2 neo cbas-core Status: MERGED +2 +1

        Activity

          People

            till Till Westmann
            michael.blow Michael Blow
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty