Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Won't Fix
-
6.0.0, 5.5.0, 5.5.1, 6.0.1
-
Centos 7
-
Untriaged
-
Release Note
-
Unknown
Description
Install Couchbase Server 5.5.0 on a centos server with IPv6 enable.
Enable IPv6 on Couchbase Server
Restart Couchbase Server
Using raw IPv6 address in browser to initialize this node with service kv, index, n1ql and fts
Verify IPv6 in Dashboard
Create a travel sample bucket
Verify all data and indexes created
Offline upgrade this node to 5.5.1
After upgrade, Couchbase Server could not access in UI with raw IPv6 address
Check config file, IPv6 was disable in Couchbase Server
Use IPv4 to open Couchbase Server, see Couchbase Server was reset to initial page.
[root@s10505-ip6 ~]# rpm -i couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm
|
Minimum RAM required : 4 GB
|
System RAM configured : 3.70 GB
|
|
|
Minimum number of processors required : 4 cores
|
Number of processors on the system : 4 cores
|
|
|
|
|
You have successfully installed Couchbase Server.
|
Please browse to http://s10505-ip6:8091/ to configure your server.
|
Please refer to http://couchbase.com for additional resources.
|
|
|
Please note that you have to update your firewall configuration to
|
allow connections to the following ports:
|
4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211,
|
11214, 11215, 18091 to 18093, and from 21100 to 21299.
|
|
|
By using this software you agree to the End User License Agreement.
|
See /opt/couchbase/LICENSE.txt.
|
|
|
[root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config
|
{ipv6, false}.
|
[root@s10505-ip6 ~]# systemctl stop couchbase-server
|
[root@s10505-ip6 ~]# sed -i '/ipv6, /c \{ipv6, true\}'. /opt/couchbase/etc/couchbase/static_config
|
[root@s10505-ip6 ~]# systemctl start couchbase-server
|
[root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config
|
{ipv6, true}.
|
[root@s10505-ip6 ~]# rpm -Uvh couchbase-server-enterprise-5.5.1-centos7.x86_64.rpm
|
Preparing... ################################# [100%]
|
Minimum RAM required : 4 GB
|
System RAM configured : 3.70 GB
|
|
|
Minimum number of processors required : 4 cores
|
Number of processors on the system : 4 cores
|
|
|
Updating / installing...
|
1:couchbase-server-5.5.1-3511 ################################# [ 50%]
|
Upgrading couchbase-server ...
|
/opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
|
Automatic mode: running without interactive questions or confirmations.
|
Analysing...
|
Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
|
Target node: ns_1@127.0.0.1
|
Done: previous node configuration is empty.
|
Cleaning up / removing...
|
2:couchbase-server-5.5.0-2958 ################################# [100%]
|
|
|
You have successfully installed Couchbase Server.
|
Please browse to http://s10505-ip6:8091/ to configure your server.
|
Please refer to http://couchbase.com for additional resources.
|
|
|
Please note that you have to update your firewall configuration to
|
allow connections to the following ports:
|
4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211,
|
11214, 11215, 18091 to 18093, and from 21100 to 21299.
|
|
|
By using this software you agree to the End User License Agreement.
|
See /opt/couchbase/LICENSE.txt.
|
|
|
[root@s10505-ip6 ~]#
|
I did try a server with both IPv6 and IPv4 enable or only IPv6 enable. In both cases, I got the same result. Offline upgrade from IPv6 enable server from 5.5.0 to 5.5.x deleted config file which reset node back to initial page.
Attachments
Issue Links
Activity
I tried on 2 nodes cluster with IPv6 enable and offline upgrade from 5.5.0 to 5.5.1 (use raw IPv6)
config.dat file is not deleted but cluster did not up in UI in server has both IPv6 and IPv4 enable or only IPv6 enable. I saw only 2 erlang processes running. After enable IPv6 on all nodes in cluster after upgrade, I could access UI again in both nodes.
Here are steps I tried.
- Create a 2 nodes cluster 5.5.0 with IPv6 enable in both node (use raw IPv6).
- Set services cluster to kv, index, n1ql and fts.
- Create travel-sample bucket in Settings page.
- Verify all data loaded to bucket and indexes created in UI.
- On both nodes, do offline upgrade to 5.5.1
- Verify upgrade is done in command output.
- Check UI on both nodes, UI fails to connect.
- Check erlang processes in nodes, see only 2 processes running.
- On node with IPv4 enable, check UI with IPv4 address, no connection either (UI down).
- Check if IPv6 is enable after upgrade, see IPv6 in both nodes are disable.
- Enable IPv6 in both nodes.
- Restart Couchbase Server, UI are up on both nodes.
- Check travel sample if it is still in cluster. Yes, travel sample still in cluster.
- Check all indexes if they are still in cluster. Yes, all indexes are still in cluster.
- Check all nodes still retain IPv6 raw address. Yes.
Here is the output when doing offline upgrade
[root@s10505-ip6 ~]# rpm -Uhv couchbase-server-enterprise-5.5.1-centos7.x86_64.rpm
|
Preparing... ################################# [100%]
|
Minimum RAM required : 4 GB
|
System RAM configured : 3.70 GB
|
|
|
Minimum number of processors required : 4 cores
|
Number of processors on the system : 4 cores
|
|
|
Updating / installing...
|
1:couchbase-server-5.5.1-3511 ################################# [ 50%]
|
Upgrading couchbase-server ...
|
/opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
|
Automatic mode: running without interactive questions or confirmations.
|
Analysing...
|
Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
|
Target node: ns_1@FD63:6F75:6368:20D4:423D:37C3:E6F7:3FAC
|
Upgrading from 2.0
|
|
|
Couchbase should not be running.
|
Please use: /etc/init.d/couchbase-server stop
|
|
|
Database dir: /opt/couchbase/var/lib/couchbase/data
|
|
|
Buckets to upgrade: travel-sample
|
|
|
Checking disk space available for buckets in directory:
|
/opt/couchbase/var/lib/couchbase/data
|
Free disk bucket space wanted: 0.0
|
Free disk bucket space available: 44531814400
|
Free disk space factor: 2.0
|
Ok.
|
|
|
Analysis complete.
|
|
|
Copying /opt/couchbase/var/lib/couchbase/config/config.dat
|
cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
|
Copying /opt/couchbase/var/lib/couchbase/ip
|
cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip
|
Ensuring bucket data directories.
|
Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/travel-sample
|
mkdir -p /opt/couchbase/var/lib/couchbase/data/travel-sample
|
Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
|
chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
|
Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
|
chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
|
|
|
Done.
|
Cleaning up / removing...
|
2:couchbase-server-5.5.0-2958 ################################# [100%]
|
|
|
You have successfully installed Couchbase Server.
|
Please browse to http://s10505-ip6:8091/ to configure your server.
|
Please refer to http://couchbase.com for additional resources.
|
|
|
Please note that you have to update your firewall configuration to
|
allow connections to the following ports:
|
4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211,
|
11214, 11215, 18091 to 18093, and from 21100 to 21299.
|
|
|
By using this software you agree to the End User License Agreement.
|
See /opt/couchbase/LICENSE.txt.
|
|
|
[root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config
|
{ipv6, false}.
|
[root@s10505-ip6 ~]# ps aux | grep beam.smp
|
couchba+ 13536 0.6 0.7 1241736 27308 ? Ssl 09:38 0:00 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -proto_dist inet_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config"
|
couchba+ 14261 104 1.4 1277468 54972 ? Ssl 09:40 0:01 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -proto_dist inet_tcp -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
|
root 14296 0.0 0.0 112660 984 pts/0 S+ 09:40 0:00 grep --color=auto beam.smp
|
[root@s10505-ip6 ~]#
|
[root@s10505-ip6 ~]#
|
[root@s10505-ip6 ~]# ps aux | grep beam.smp
|
couchba+ 13536 0.5 0.6 1241224 27124 ? Ssl 09:38 0:00 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -proto_dist inet_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config"
|
root 14478 0.0 0.0 112660 988 pts/0 S+ 09:40 0:00 grep --color=auto beam.smp
|
[root@s10505-ip6 ~]# sed -i '/ipv6, /c \{ipv6, true\}'. /opt/couchbase/etc/couchbase/static_config
|
[root@s10505-ip6 ~]# systemctl stop couchbase-server
|
[root@s10505-ip6 ~]# systemctl start couchbase-server
|
[root@s10505-ip6 ~]# ps aux | grep beam.smp
|
couchba+ 14768 0.4 0.7 1259476 30860 ? Ssl 09:41 0:02 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@::1 -proto_dist inet6_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config"
|
couchba+ 14866 9.0 3.5 1600656 138800 ? Ssl 09:41 0:54 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -proto_dist inet6_tcp -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
|
couchba+ 14902 0.4 1.2 1035844 49140 ? Ssl 09:41 0:02 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -pa /opt/couchbase/lib/erlang/lib/appmon-2.1.14.2/ebin /opt/couchbase/lib/erlang/lib/asn1-2.0.4/ebin /opt/couchbase/lib/erlang/lib/compiler-4.9.4/ebin /opt/couchbase/lib/erlang/lib/cosEvent-2.1.14/ebin /opt/couchbase/lib/erlang/lib/cosEventDomain-1.1.13/ebin /opt/couchbase/lib/erlang/lib/cosFileTransfer-1.1.15/ebin /opt/couchbase/lib/erlang/lib/cosNotification-1.1.20/ebin /opt/couchbase/lib/erlang/lib/cosProperty-1.1.16/ebin /opt/couchbase/lib/erlang/lib/cosTime-1.1.13/ebin /opt/couchbase/lib/erlang/lib/cosTransactions-1.2.13/ebin /opt/couchbase/lib/erlang/lib/crypto-3.2/ebin /opt/couchbase/lib/erlang/lib/dialyzer-2.6.1/ebin /opt/couchbase/lib/erlang/lib/diameter-1.5/ebin /opt/couchbase/lib/erlang/lib/edoc-0.7.12.1/ebin /opt/couchbase/lib/erlang/lib/eldap-1.0.2/ebin /opt/couchbase/lib/erlang/lib/erl_docgen-0.3.4.1/ebin /opt/couchbase/lib/erlang/lib/erl_interface-3.7.15 /opt/couchbase/lib/erlang/lib/erts-5.10.4.0.0.1/ebin /opt/couchbase/lib/erlang/lib/et-1.4.4.5/ebin /opt/couchbase/lib/erlang/lib/eunit-2.2.6/ebin /opt/couchbase/lib/erlang/lib/gs-1.5.15.2/ebin /opt/couchbase/lib/erlang/lib/hipe-3.10.2.2/ebin /opt/couchbase/lib/erlang/lib/ic-4.3.4/ebin /opt/couchbase/lib/erlang/lib/inets-5.9.8/ebin /opt/couchbase/lib/erlang/lib/jinterface-1.5.8 /opt/couchbase/lib/erlang/lib/mnesia-4.11/ebin /opt/couchbase/lib/erlang/lib/orber-3.6.26.1/ebin /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/ebin /opt/couchbase/lib/erlang/lib/otp_mibs-1.0.8/ebin /opt/couchbase/lib/erlang/lib/parsetools-2.0.10/ebin /opt/couchbase/lib/erlang/lib/percept-0.8.8.2/ebin /opt/couchbase/lib/erlang/lib/pman-2.7.1.4/ebin /opt/couchbase/lib/erlang/lib/public_key-0.21/ebin /opt/couchbase/lib/erlang/lib/reltool-0.6.4.1/ebin /opt/couchbase/lib/erlang/lib/runtime_tools-1.8.13/ebin /opt/couchbase/lib/erlang/lib/sasl-2.3.4/ebin /opt/couchbase/lib/erlang/lib/snmp-4.25/ebin /opt/couchbase/lib/erlang/lib/ssh-3.0/ebin /opt/couchbase/lib/erlang/lib/ssl-5.3.3/ebin /opt/couchbase/lib/erlang/lib/syntax_tools-1.6.13/ebin /opt/couchbase/lib/erlang/lib/toolbar-1.4.2.3/ebin /opt/couchbase/lib/erlang/lib/tools-2.6.13/ebin /opt/couchbase/lib/erlang/lib/tv-2.1.4.10/ebin /opt/couchbase/lib/erlang/lib/typer-0.9.5/ebin /opt/couchbase/lib/erlang/lib/webtool-0.8.9.2/ebin /opt/couchbase/lib/erlang/lib/xmerl-1.3.6/ebin /opt/couchbase/lib/couchdb/plugins/gc-couchbase-1.0.0/ebin /opt/couchbase/lib/couchdb/plugins/vtree-0.1.0/ebin /opt/couchbase/lib/couchdb/plugins/wkb-1.2.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_dcp-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_index_merger-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_set_view-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/erlang-oauth/ebin /opt/couchbase/lib/couchdb/erlang/lib/etap/ebin /opt/couchbase/lib/couchdb/erlang/lib/lhttpc-1.3/ebin /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/mochiweb-1.4.1/ebin /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/ebin /opt/couchbase/lib/ns_server/erlang/lib/ale/ebin /opt/couchbase/lib/ns_server/erlang/lib/gen_smtp/ebin /opt/couchbase/lib/ns_server/erlang/lib/meck/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_babysitter/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_couchdb/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_server/ebin /opt/couchbase/lib/ns_server/erlang/lib/triq/ebin /opt/couchbase/lib/erlang/lib/stdlib-1.19.4/ebin /opt/couchbase/lib/erlang/lib/kernel-2.16.4/ebin . -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -setcookie xcrfzuwgxrcuedvt -name couchdb_ns_1@::1 -smp enable -kernel error_logger false -sasl sasl_error_logger false -nouser -hidden -proto_dist inet6_tcp -run child_erlang child_start ns_couchdb
|
root 15196 0.0 0.0 112664 988 pts/0 S+ 09:51 0:00 grep --color=auto beam.smp
|
[root@s10505-ip6 ~]#
|
I will collect logs and upload here soon
So the work around for 2 and more nodes cluster is re-enable IPv6 and restart Couchbase Server after offline upgrade.
I uploaded 2 logs from 2 IPv6 nodes cluster.
fd63.6f75.6368.20d4.423d.37c3.e6f7.3fac-20181010-1021-diag.zip
fd63.6f75.6368.20d4.9e7.4c7e.6879.d510-20181010-1022-diag.zip
This is obviously a bit unfortunate, however, given that 5.5.0 and 5.5.1 are shipped pretty long ago and we have evidence that most folks are on 5.5.2 and later and this only affects single node IPv6, we are planning to document this limitation.
Ajit to link to doc ticket.
Bulk closing invalid, duplicate, won't fix bugs. Please feel free to reopen them
I will try 2 IPv6 nodes cluster tomorrow if I could reproduce this issue.