Critical Description
steps:
1. there are 3 nodes 1.8.0r in cluster with 5 different buckets
2. upgrade master 10.3.121.112 without stopping it (sudo rpm -U couchbase-server-community_x86_64_2.0.0-1797-rel.rpm )
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1797-rel.rpm.manifest.xml
result: cluster is alive, but the master is in pending state
Current master is older (before 2.0) and I'll try to takeover (repeated 1 times) mb_master000 ns_1@10.3.121.112 18:32:46 - Thu Oct 4, 2012
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.121.112 18:31:56 - Thu Oct 4, 2012
Node 'ns_1@10.3.121.114' saw that node 'ns_1@10.3.121.112' came up. ns_node_disco004 ns_1@10.3.121.114 18:31:47 - Thu Oct 4, 2012
[ns_server:debug,2012-10-04T8:31:46.976,ns_1@10.3.121.112:'couch_stats_reader-sasl':couch_stats_reader:vbuckets_aggregation_loop:117]Failed to open vbucket: 124 (
{not_found,no_db_file}). Ignoring[ns_server:debug,2012-10-04T8:31:46.977,ns_1@10.3.121.112:'couch_stats_reader-sasl':couch_stats_reader:vbuckets_aggregation_loop:117]Failed to open vbucket: 125 ({not_found,no_db_file}
). Ignoring
[ns_server:debug,2012-10-04T8:31:46.977,ns_1@10.3.121.112:'couch_stats_reader-sasl':couch_stats_reader:vbuckets_aggregation_loop:117]Failed to open vbucket: 126 (
[ns_server:debug,2012-10-04T8:31:46.977,ns_1@10.3.121.112:'couch_stats_reader-sasl':couch_stats_reader:vbuckets_aggregation_loop:117]Failed to open vbucket: 127 ({not_found,no_db_file}
). Ignoring
[ns_server:info,2012-10-04T8:31:46.995,ns_1@10.3.121.112:ns_port_memcached:ns_port_server:log:169]memcached<0.686.0>: Thu Oct 4 08:31:46.792889 PDT 3: Trying to connect to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.794067 PDT 3: Connected to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.803198 PDT 3: Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"
memcached<0.686.0>: Thu Oct 4 08:31:46.803515 PDT 3: Extension support isn't implemented in this version of bucket_engine
memcached<0.686.0>: Thu Oct 4 08:31:46.804070 PDT 3: Failed to load mutation log, falling back to key dump
memcached<0.686.0>: Thu Oct 4 08:31:46.804388 PDT 3: metadata loaded in 2409 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.804677 PDT 3: warmup completed in 2638 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.820648 PDT 3: Trying to connect to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.821185 PDT 3: Connected to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.828962 PDT 3: Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"
memcached<0.686.0>: Thu Oct 4 08:31:46.829211 PDT 3: Failed to load mutation log, falling back to key dump
memcached<0.686.0>: Thu Oct 4 08:31:46.829439 PDT 3: Extension support isn't implemented in this version of bucket_engine
memcached<0.686.0>: Thu Oct 4 08:31:46.829708 PDT 3: metadata loaded in 1310 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.829921 PDT 3: warmup completed in 1424 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.831783 PDT 3: Trying to connect to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.832142 PDT 3: Connected to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.835130 PDT 3: Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"
memcached<0.686.0>: Thu Oct 4 08:31:46.835280 PDT 3: Extension support isn't implemented in this version of bucket_engine
memcached<0.686.0>: Thu Oct 4 08:31:46.835426 PDT 3: Failed to load mutation log, falling back to key dump
memcached<0.686.0>: Thu Oct 4 08:31:46.835583 PDT 3: metadata loaded in 830 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.835936 PDT 3: warmup completed in 1070 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.840392 PDT 3: Trying to connect to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.840701 PDT 3: Connected to mccouch: "localhost:11213"
memcached<0.686.0>: Thu Oct 4 08:31:46.847749 PDT 3: Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"
memcached<0.686.0>: Thu Oct 4 08:31:46.847966 PDT 3: Failed to load mutation log, falling back to key dump
memcached<0.686.0>: Thu Oct 4 08:31:46.848338 PDT 3: metadata loaded in 1298 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.848676 PDT 3: warmup completed in 1586 usec
memcached<0.686.0>: Thu Oct 4 08:31:46.848889 PDT 3: Extension support isn't implemented in this version of bucket_engine