Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19153

[3.1.5] unable to add back removed node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 3.1.5
    • 3.1.5
    • couchbase-bucket
    • None

    Description

      3.1.5 - 8x2 Cluster with xdcr and views

      I removed a node, then was unable to add it back or log in to the node itself:

      Failed to add node 172.23.106.213:8091 to cluster. Prepare join failed. Timeout connecting to "172.23.106.213" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.
      

      the debug logs repeat the following message

      [ns_server:debug,2016-04-11T13:51:15.242,ns_1@172.23.106.213:ns_heart_slow_status_updater<0.1024.0>:cluster_logs_collection_task:maybe_build_cluster_logs_task:43]Ignoring exception trying to read cluster_logs_collection_task_status table: error:badarg
      [ns_server:debug,2016-04-11T13:51:20.237,ns_1@172.23.106.213:ns_heart_slow_status_updater<0.1024.0>:ns_heart:current_status_slow_inner:259]Ignoring failure to grab system stats:
      {'EXIT',{noproc,{gen_server,call,
                                  [{'stats_reader-@system','ns_1@172.23.106.213'},
                                   {latest,"minute"}]}}}
       
      [ns_server:debug,2016-04-11T13:51:20.241,ns_1@172.23.106.213:ns_heart_slow_status_updater<0.1024.0>:cluster_logs_collection_task:maybe_build_cluster_logs_task:43]Ignoring exception trying to read cluster_logs_collection_task_status table: error:badarg
      [ns_server:debug,2016-04-11T13:51:25.239,ns_1@172.23.106.213:ns_heart_slow_status_updater<0.1024.0>:ns_heart:current_status_slow_inner:259]Ignoring failure to grab system stats:
      {'EXIT',{noproc,{gen_server,call,
                                  [{'stats_reader-@system','ns_1@172.23.106.213'},
                                   {latest,"minute"}]}}}
       
      [ns_server:debug,2016-04-11T13:51:25.243,ns_1@172.23.106.213:ns_heart_slow_status_updater<0.1024.0>:cluster_logs_collection_task:maybe_build_cluster_logs_task:43]Ignoring exception trying to read cluster_logs_collection_task_status table: error:badarg
      

      Also noticed from the ui console that requests to /pools/default/tasks is failing:
      curl http://172.23.106.213:8091/pools/default/tasks
      ["Unexpected server error, request logged."]

      [ns_server:error,2016-04-11T13:41:35.312,ns_1@172.23.106.213:<0.48.1>:menelaus_web:loop:170]Server error during processing: ["web request failed",
                                       {path,"/pools/default/tasks"},
                                       {type,exit},
                                       {what,
                                        {noproc,
                                         {gen_server,call,
                                          [xdc_rdoc_replication_srv,
                                           {foreach_doc,
                                            #Fun<xdc_rdoc_replication_srv.2.4398899>},
                                           infinity]}}},
                                       {trace,
                                        [{gen_server,call,3,
                                          [{file,"gen_server.erl"},{line,188}]},
                                         {xdc_rdoc_replication_srv,
                                          find_all_replication_docs,1,
                                          [{file,"src/xdc_rdoc_replication_srv.erl"},
                                           {line,215}]},
                                         {ns_doctor,build_tasks_list,2,
                                          [{file,"src/ns_doctor.erl"},{line,198}]},
                                         {menelaus_web,handle_tasks,2,
                                          [{file,"src/menelaus_web.erl"},
                                           {line,2553}]},
                                         {request_throttler,do_request,3,
                                          [{file,"src/request_throttler.erl"},
                                           {line,59}]},
                                         {menelaus_web,loop,2,
                                          [{file,"src/menelaus_web.erl"},
                                           {line,149}]},
                                         {mochiweb_http,headers,5,
                                          [{file,
                                            "/home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/couchdb/src/mochiweb/mochiweb_http.erl"},
                                           {line,94}]},
                                         {proc_lib,init_p_do_apply,3,
                                          [{file,"proc_lib.erl"},{line,239}]}]}]
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              tommie Tommie McAfee (Inactive)
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty