Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48221

Node rename during engageCluster fails with chronicle conflict

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.2-6610

      Test: ./testrunner -i /tmp/testexec.127646.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P0,index_type=scorch,fts_quota=990,disableFileTransferRebalance=False,enforce_tls=True,use_https=True -t fts.moving_topology_fts.MovingTopFTS.swap_rebalance_kv_during_index_building,items=30000,cluster=D,F,replicas=0,GROUP=P0

      Test steps:

      • Setup 2 node cluster with 172.23.123.47: kv, 172.23.123.44: fts
      • Enforce TLS: Set up node to node encryption - status = enable and clusterEncryptionLevel = strict
      • Create default bucket and default_index_1 fts index
      • Load 30000 docs
      • Swap rebalance kv node, which is successful

        Starting swap-rebalance [remove_node:[ip:172.23.123.47 port:8091 ssh_username:root]] -> [add_node:[ip:172.23.123.42 port:8091 ssh_username:root]] at C1 cluster 172.23.123.47
        

      • Validate index count with bucket count
      • Swap Rebalance fts node

        Starting swap-rebalance [remove_node:[ip:172.23.123.44 port:8091 ssh_username:root]] -> [add_node:[ip:172.23.123.56 port:8091 ssh_username:root]] at C1 cluster 172.23.123.44
        

      • Rebalance fails with below:

        [2021-08-27 19:48:03,490] - [rest_client:1055] ERROR - POST https://172.23.123.44:18091/controller/addNode body: hostname=https%3A%2F%2F172.23.123.56%3A18091&user=Administrator&password=password&services=fts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["Prepare join failed. Got HTTP status 500 from REST call post to https://172.23.123.56:18091/engageCluster2. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\""]' auth: Administrator:password
        

      From the logs of 172.23.123.56

      =========================NOTICE REPORT=========================
      TLS client: In state certify at ssl_handshake.erl:1783 generated CLIENT ALERT: Fatal - Handshake Failure
       - {bad_cert,hostname_check_failed}
      [error_logger:info,2021-08-27T19:48:03.445-07:00,ns_1@172.23.123.56:net_kernel<0.24388.2>:ale_error_logger_handler:do_log:101]
      =========================NOTICE REPORT=========================
      {net_kernel,
          {'EXIT',<0.24398.2>,
              {ssl_connect_failed,
                  {127,0,0,1},
                  21150,
                  {error,
                      {tls_alert,
                          {handshake_failure,
                              "TLS client: In state certify at ssl_handshake.erl:1783 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}}}}}
      

      Note: Seen this in Automated run. But couldn't reproduce in local run.

      Attachments

        1. test_1.zip
          14.46 MB
        2. test_8.zip
          30.24 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty