Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61160

[Analytics] : Rebalance exited with reason {service_rebalance_failed,cbas,{{badmatch,{error,{bad_nodes,cbas,set_service_manager,[{'ns_1@172.23.136.185',{exit,{{linked_process_died,<0.26915.1>,{'ns_1@172.23.136.185',{no_connection,"cbas-service_api"}}},

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1.Automatically regenerated cluster certificate on all nodes

      $/cygdrive/c/Program\ Files/Couchbase/Server/bin/couchbase-cli.exe ssl-manage -c http://172.23.136.182 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem 

      2. Created a private key for the cluster and self-signed it (root cert)

      $openssl genrsa  -out /tmp/newcerts20/ca.key 1024  
       
      $openssl req -new -x509  -days 3650 -sha256 -key /tmp/newcerts20/ca.key -out /tmp/newcerts20/ca.pem -subj '/C=UA/O=My Company/CN=My Company Root CA' 

      3. Created an intermediate certificate signed using the root cert 

      $openssl genrsa  -out /tmp/newcerts20/int.key 1024 
       
      $openssl req -new -key /tmp/newcerts20/int.key -out /tmp/newcerts20/int.csr -subj '/C=UA/O=My Company/CN=My Company Intermediate CA'
       
      $openssl x509 -req -in /tmp/newcerts20/int.csr -CA /tmp/newcerts20/ca.pem -CAkey /tmp/newcerts20/ca.key -CAcreateserial -CAserial /tmp/newcerts20/rootCA.srl -extfile ./pytests/security/v3_ca.ext -out /tmp/newcerts20/int.pem -days 365 -sha256
       

      4. Generated a certificate for each node and signed with the intermediate certificate

      $openssl genrsa  -out /tmp/newcerts20/172.23.136.182.key 1024
       
      $openssl req -new -key /tmp/newcerts20/172.23.136.182.key -out /tmp/newcerts20/172.23.136.182.csr -config ./pytests/security/clientconf3.conf 
       
      $openssl x509 -req -in /tmp/newcerts20/172.23.136.182.csr -CA /tmp/newcerts20/int.pem -CAkey /tmp/newcerts20/int.key -CAcreateserial -CAserial /tmp/newcerts20/intermediateCA.srl -out /tmp/newcerts20/172.23.136.182.pem -days 365 -sha256 -extfile ./pytests/security/clientconf3.conf -extensions req_ext
       
      $cat /tmp/newcerts20/172.23.136.182.pem /tmp/newcerts20/int.pem /tmp/newcerts20/ca.pem > /tmp/newcerts20/long_chain172.23.136.182.pem 

      5. Uploaded the root cert and the node cert(eg. long_chain172.23.136.182.pem) onto each node

      6. Added nodes 172.23.136.183, 172.23.136.184, 172.23.136.185 onto 172.23.136.182

      7. Started a rebalance - Rebalance fails

      2024-03-04T07:41:36.769-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.136.185) - Rebalance exited with reason {service_rebalance_failed,cbas,                              {{badmatch,                                {error,                                 {bad_nodes,cbas,set_service_manager,                                  [{'ns_1@172.23.136.185',                                    {exit,                                     {{linked_process_died,<0.26915.1>,                                       {'ns_1@172.23.136.185',                                        {no_connection,"cbas-service_api"}}},                                      {gen_server,call,                                       [{'service_agent-cbas',                                         'ns_1@172.23.136.185'},                                        {set_service_manager,<0.26958.1>},                                        infinity]}}}}]}}},                               [{service_manager,set_service_manager,1,                                 [{file,"src/service_manager.erl"},                                  {line,188}]},                                {service_manager,run_op,1,                                 [{file,"src/service_manager.erl"},                                  {line,146}]},                                {proc_lib,init_p,3,                                 [{file,"proc_lib.erl"},{line,225}]}]}}.Rebalance Operation Id = 49ed1afd67c1dbcfe904bb3e66af94a3 

       


       

      Testrunner script to reproduce

      ./testrunner -i /tmp/testexec.9861.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False,get-cbcollect-info=True,get-cbcollect-info=True -t security.x509tests.x509tests.test_add_node_with_cert_diff_services,services_in=kv-kv-cbas,default_bucket=False,SSLtype=openssl,client_cert_state=enable,bucket_storage=couchstore,setup_once=True,GROUP=P0;P0_SERVICE
      

      Job name : windows19-os_certify-security-x509

      Job ref : http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-2176/jenkins_logs/test_suite_executor-dynvm/19890/

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Abhijeeth.Nuthan Abhijeeth Nuthan
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty