I'm seeing intermittent problems when building a cluster of Membase servers. Sometimes adding an additional server to a cluster will cause another server to be marked as unhealthy. Other times the additional server will function correctly but all others will appear to be in setup mode. This only happens some times, so it's tricky to nail down properly.
Steps to reproduce this are in this forum thread: http://www.couchbase.org/forums/thread/server-marked-unhealthy-after-adding-additional-server-cluster
In the most recent test run, I spun up 3 m2.xlarge instances as described in the forum post. On adding the third server to the cluster (from the admin web interface on the first server), the admin interface refreshed and showed the setup dialog. When I browsed to the second server it was also in setup mode. The third server was configured correctly and appeared to be running in its own cluster of one machine. I've attached the output from /logs on all three machines.
(Forgive me if I have the wrong component selected, I'm not quite sure which one is causing me to see this issue.)
|For Gerrit Dashboard: &For+MB-4476=message:MB-4476|
|14062,3||reset node's cookie before joining cluster.MB-4476||ns_server||Status: ABANDONED||+2||+1|
|14186,1||reset node's cookie before joining cluster.MB-4476||ns_server||Status: MERGED||+2||+1|
|14229,1||Merge branch 'branch-181' into branch-18||ns_server||Status: MERGED||+2||+1|
|14233,1||Merge remote-tracking branch 'origin/branch-18' into master||ns_server||Status: MERGED||+2||+1|