Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.2.0
-
Security Level: Public
-
None
Description
Setup 2 clusters,
Load 500K on cluster1
Start replicating from Cluster1 to Cluster2 ( 3 to 3 nodes)
Seeing frequent memcached disconnect errors messages on 2 out of 3 nodes on destination cluster.
The 2 nodes go into pending state.
- Front end load on source ~ 20K sets, replicating @ 10k sets to remote cluster
Able to repro this very frequently ( three times) so far on the same set of machines.
Now moving to another set of machines to verify.
- Also seen in the past ( 1 week back), node going down eventually and had to be removed from the cluster.
Seeing this typically w/ only XDCR remote cluster so far.
Errors from the UI logs
Control connection to memcached on 'ns_1@172.23.105.47' disconnected: {{badmatch,
{error,
timeout}},
[
,
{mc_client_binary, select_bucket, 2},
{ns_memcached, ensure_bucket, 2},
{ns_memcached, handle_info, 2},
{gen_server, handle_msg, 5},
{ns_memcached, init, 1},
{gen_server, init_it, 6},
{proc_lib, init_p_do_apply, 3}]}