[system test] online upgrade failed due to "Conflicting configuration changes to field {node,'ns_1@10.6.2.39', config_version}"

Cluster information:

• 8 centos 6.2 64bit server with 4 cores CPU
• Each server has 32 GB RAM and 400 GB SSD disk.
• 24.8 GB RAM for couchbase server at each node
• SSD disk format ext4 on /data
• Each server has its own SSD drive, no disk sharing with other server.
• Create cluster with 6 nodes installed couchbase server 1.8.1-937
• Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
• Load 16 million items to each bucket with key size from 512 to 1024 bytes

10.6.2.37
10.6.2.38
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43

• Mutate 16 million items with size from 1024 to 1500 bytes
• Install couchbase server 2.0.0-1885 on node 44 and 45

• Link to manifest file of build 2.0.0-1885 http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1885-rel.deb.manifest.xml

• Do online upgrade on cluster from 1.8.1 to 2.0.0 by using swap rebalance.
• Add node 44 and remove node 39. Rebalance passed.
• Add node 45 and remove node 40. Rebalance passed

• Uninstall couchbase server 1.8.1 on node 39. Then delete dir /opt/couchbase and all files/dir in /data/
• Install couchbase server 2.0.0-1885 on node 39. Go through initial setup and set data/view path to /data/
• Add node 39 back to cluster to do swap rebalance. Add node failed with error:

Conflicting configuration changes to field

:
{[{'ns_1@10.6.2.39',{1,63518277795}},{'ns_1@127.0.0.1',{5,63518276395}}],
[{'_vclock',[{'ns_1@10.6.2.39',{1,63518277795}},
{'ns_1@127.0.0.1',{5,63518276395}}]}|

]} and
{[{'ns_1@10.6.2.39',{1,63518255426}},{'ns_1@127.0.0.1',{5,63518255102}}],
[{'_vclock',[{'ns_1@10.6.2.39',{1,63518255426}},
{'ns_1@127.0.0.1',{5,63518255102}}]}|

]}, choosing the former, which looks newer.

• Then web UI on node 39 is down. Go to node 39, couchbase server is still running

[root@localhost ~]# service couchbase-server status
couchbase-server is running
[root@localhost ~]# ps aux | grep couchbase
root 4169 0.0 0.0 103236 868 pts/0 S+ 10:43 0:00 grep couchbase
498 8808 0.0 0.0 10824 596 ? S Oct23 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon
498 8821 3.8 0.6 1756544 221768 ? Sl Oct23 27:35 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K true – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -noshell -noinput -noshell -noinput -run ns_bootstrap – -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.node" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
498 8850 0.0 0.0 4052 584 ? Ss Oct23 0:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup
498 8851 0.0 0.0 4048 436 ? Ss Oct23 0:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
498 8853 0.0 0.0 38972 2132 ? Ss Oct23 0:00 /opt/couchbase/lib/erlang/lib/ssl-4.1.6/priv/bin/ssl_esock

Link to collect info of all node https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-ci-upgrade-181-to-200-20121023-232544.tgz