Description
Build
-------
3.5.0-1270
- In previous test, we rebooted .62 (hard shutdown of Couchbase server) and firewall was disabled afterwards, replication happened fine. We then removed .62 from .57 cluster
- In next test, we tried to add .62 to .57 again, .62 was unresponsive and requests timed-out.
Seeing errors like -
Error adding node: 172.23.105.62 to the cluster:172.23.105.57 - ["Prepare join failed. Timeout connecting to \"172.23.105.62\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers."]
Easily reproducible after running -
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-logs=False,stop-on-failure=False,replication_type=xmem,enable_goxdcr=True,checkpoint_interval=3600 -t xdcr.biXDCR.bidirectional.replication_while_rebooting_a_non_master_src_dest_node,items=10000,rdirection=bidirection,sasl_buckets=1,ctopology=chain
Attaching logs.
[ns_server:error,2015-02-17T16:36:51.798,ns_1@172.23.105.62:<0.28306.0>:menelaus_web:loop:182]Server error during processing: ["web request failed",
,
,
{what,
{timeout,
{gen_server,call,
[ns_cluster,
{engage_cluster,
[
,
{<<"availableStorage">>,
{struct,
[{<<"hdd">>,
[{struct,
[
,
,
]},
{struct,
[
,
,
]},
{struct,
[
,
,
]},
{struct,
[
,
,
]}]}]}},
,
{<<"storageTotals">>,
{struct,
[{<<"ram">>,
{struct,
[
,
,
,
,
{<<"quotaUsedPerNode">>,0},
{<<"quotaTotalPerNode">>, 8508145664}]}},
{<<"hdd">>,
{struct,
[{<<"total">>,36226977792},
{<<"quotaTotal">>,36226977792},
{<<"used">>,17026679562},
{<<"usedByData">>,0}
,
]}}]}},
{<<"storage">>,
{struct,
[
,
{<<"hdd">>,
[{struct,
[
,
,
,
]}]}]}},
{<<"systemStats">>,
{struct,
[
,
,
,
,
]}},
{<<"interestingStats">>,{struct,[]}},
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
{<<"ports">>,
{struct,
[
,
,
,
,
]}},
]},
30000]}}},
{trace,
[{gen_server,call,3,
[
,
{line,188}]},
{menelaus_web,handle_engage_cluster2,1,
[
{line,871}]},
{request_throttler,do_request,3,
[{file,"src/request_throttler.erl"},
{line,59}]},
{menelaus_web,loop,2,
[{file,"src/menelaus_web.erl"}
,
]},
{mochiweb_http,headers,5,
[
,
]},
{proc_lib,init_p_do_apply,3,
[
,
{line,239}]}]}]