Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51510

Node add fails with connection_closed error

    XMLWordPrintable

Details

    • Bug
    • Resolution: User Error
    • Critical
    • Morpheus
    • Morpheus
    • ns_server
    • 7.2.0-1018
    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

       

      Build: 7.2.0-1018

      Trying to add node fails with the following error

      [user:info,2022-03-18T02:16:38.124-07:00,ns_1@172.23.105.155:<0.30744.0>:ns_cluster:add_node_to_group:87]
      Failed to add node 172.23.105.211:18091 to cluster. Prepare join failed. Error connection_closed happened during REST call post to https://172.23.105.211:18091/engageCluster2.

      From .155 error.log

       

      [ns_server:error,2022-03-18T02:39:13.549-07:00,ns_1@172.23.105.155:<0.2939.0>:menelaus_util:reply_server_error_before_close:210]Server error during processing: ["web request failed",
      {path,"/controller/addNode"},
        {method,'POST'}, {type,exit},
        {what, {
          {{{{badmatch,{error,enoent}},
                 [{ns_ssl_services_setup, save_node_certs,6,
                    [{file, "src/ns_ssl_services_setup.erl"}, {line,781}]},
                  {ns_ssl_services_setup,  maybe_generate_node_certs,0,
                    [{file, "src/ns_ssl_services_setup.erl"}, {line,734}]},
                  {ns_ssl_services_setup,handle_info,2,
                    [{file, "src/ns_ssl_services_setup.erl"}, {line,590}]},
                  {gen_server,try_dispatch,4,
                    [{file,"gen_server.erl"},{line,695}]},
                  {gen_server,handle_msg,6,
                    [{file,"gen_server.erl"},{line,771}]},
                  {proc_lib,init_p_do_apply,3,
                    [{file,"proc_lib.erl"},{line,226}]}]},
                  {gen_server,call,
                    [ns_ssl_services_setup,ping,300000]}},
                  {gen_server,call,  [dist_manager,
                    {adjust_my_address,"172.23.105.155",
                     false,#Fun<ns_cluster.8.8996322>,
                     #Fun<ns_cluster.13.8996322>},
                  infinity]}},
            {gen_server,call,  [ns_cluster,
              {add_node_to_group,https, "172.23.105.211",18091,
               {"Administrator","password"}, undefined,  [kv]},
              240000]}}},
        {trace,
         [{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},
          {ns_cluster,add_node_to_group,6,
           [{file,"src/ns_cluster.erl"},{line,83}]},
          {menelaus_web_cluster, '-do_handle_add_node/2-fun-1-',8,
           [{file,"src/menelaus_web_cluster.erl"}, {line,764}]},
          {menelaus_util,survive_web_server_restart,
           1, [{file,"src/menelaus_util.erl"}, {line,754}]},
          {request_tracker,request,2,
           [{file,"src/request_tracker.erl"}, {line,40}]},
          {menelaus_util,handle_request,2,
           [{file,"src/menelaus_util.erl"}, {line,221}]},
          {mochiweb_http,headers,6,
           [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"}, {line,153}]},
          {proc_lib,init_p_do_apply,3,
           [{file,"proc_lib.erl"},{line,226}]}]}]
       
      [ns_server:error,2022-03-18T02:39:13.559-07:00,ns_1@172.23.105.155:prometheus_cfg<0.414.0>:prometheus_cfg:terminate:529]Terminate: shutdown
      [ns_server:error,2022-03-18T02:39:24.046-07:00,ns_1@172.23.105.155:prometheus_cfg<0.3306.0>:prometheus_cfg:terminate:529]Terminate: shutdown
      [ns_server:error,2022-03-18T02:39:27.052-07:00,ns_1@172.23.105.155:<0.3597.0>:prometheus:post_async:192]Prometheus http request failed:
      URL: http://127.0.0.1:9123/api/v1/query
      

       

      Triggering cbcollect-info through UI results in ns_server crash following by UI unreachable.

       

      Attachments

        1. 105.zip
          9.86 MB
        2. 155_2.zip
          5.16 MB
        3. 211_2.zip
          4.23 MB
        4. 211.zip
          10.12 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty