Details
Description
Setup a large cluster(14 nodes)
Cluster is in DGM ( Most of the nodes are nearly full).
Node 232 which was nearly 100% disk full, and was failed over. It took around 8 minutes for this node to fail-over.
Is this time-frame for failover completion expected?
Output from the cbstats :
-----------------------------------
[root@rvm-0102 ~]# tail -f /opt/couchbase/var/lib/couchbase/logs/info.1
[stats:error] [2012-05-10 13:41:17] [ns_1@10.3.121.206:<0.6617.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:20] [ns_1@10.3.121.206:<0.15129.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:20] [ns_1@10.3.121.206:<0.6617.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:23] [ns_1@10.3.121.206:<0.6617.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:26] [ns_1@10.3.121.206:<0.6617.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:29] [ns_1@10.3.121.206:<0.15129.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:30] [ns_1@10.3.121.206:<0.6617.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:40] [ns_1@10.3.121.206:<0.13922.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:41:49] [ns_1@10.3.121.206:<0.15128.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:42:00] [ns_1@10.3.121.206:<0.13922.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:42:09] [ns_1@10.3.121.206:<0.15128.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[stats:error] [2012-05-10 13:42:11] [ns_1@10.3.121.206:<0.15128.26>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@10.3.121.232']
[ns_server:info] [2012-05-10 13:42:12] [ns_1@10.3.121.206:ns_config_rep:ns_config_rep:do_pull:310] Pulling config from: 'ns_1@10.3.121.232'
Attaching the diags from the cluster