Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0
-
Security Level: Public
-
centos 6.2 64bit upgrade from build 1.8.1 to 2.0.0-1885
Description
Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- 24.8 GB RAM for couchbase server at each node
- SSD disk format ext4 on /data
- Each server has its own SSD drive, no disk sharing with other server.
- Create cluster with 6 nodes installed couchbase server 1.8.1-937
- Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
- Load 16 million items to each bucket with key size from 512 to 1024 bytes
10.6.2.37
10.6.2.38
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43
- Mutate 16 million items with size from 1024 to 1500 bytes
- Install couchbase server 2.0.0-1885 on node 44 and 45
- Link to manifest file of build 2.0.0-1885 http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1885-rel.deb.manifest.xml
- Do online upgrade on cluster from 1.8.1 to 2.0.0 by using swap rebalance.
- Add node 44 and remove node 39. Rebalance passed.
- Add node 45 and remove node 40. Rebalance passed
- Uninstall couchbase server 1.8.1 on node 39. Then delete dir /opt/couchbase and all files/dir in /data/
- Install couchbase server 2.0.0-1885 on node 39. Go through initial setup and set data/view path to /data/
- Add node 39 back to cluster to do swap rebalance. Add node failed with error:
Conflicting configuration changes to field
{node,'ns_1@10.6.2.39', config_version}:
{[{'ns_1@10.6.2.39',{1,63518277795}},{'ns_1@127.0.0.1',{5,63518276395}}],
[{'_vclock',[{'ns_1@10.6.2.39',{1,63518277795}},
{'ns_1@127.0.0.1',{5,63518276395}}]}|
]} and
{[{'ns_1@10.6.2.39',{1,63518255426}},{'ns_1@127.0.0.1',{5,63518255102}}],
[{'_vclock',[{'ns_1@10.6.2.39',{1,63518255426}},
{'ns_1@127.0.0.1',{5,63518255102}}]}|
]}, choosing the former, which looks newer.
- Then web UI on node 39 is down. Go to node 39, couchbase server is still running
[root@localhost ~]# service couchbase-server status
couchbase-server is running
[root@localhost ~]# ps aux | grep couchbase
root 4169 0.0 0.0 103236 868 pts/0 S+ 10:43 0:00 grep couchbase
498 8808 0.0 0.0 10824 596 ? S Oct23 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon
498 8821 3.8 0.6 1756544 221768 ? Sl Oct23 27:35 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K true – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -noshell -noinput -noshell -noinput -run ns_bootstrap – -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.node" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
498 8850 0.0 0.0 4052 584 ? Ss Oct23 0:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup
498 8851 0.0 0.0 4048 436 ? Ss Oct23 0:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
498 8853 0.0 0.0 38972 2132 ? Ss Oct23 0:00 /opt/couchbase/lib/erlang/lib/ssl-4.1.6/priv/bin/ssl_esock
Link to collect info of all node https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-ci-upgrade-181-to-200-20121023-232544.tgz
Attachments
For Gerrit Dashboard: MB-7000 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
19211,23 | MB-7000: An okay commit. | master | ep-engine | Status: ABANDONED | -2 | 0 |
21999,2 | MB-7000 Fake per node memcached config vclock. | master | ns_server | Status: MERGED | +2 | +1 |