Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31598

One node IPv6 server does offline upgrade from 5.5.0 to 5.5.1 reset node back to initial page

    XMLWordPrintable

Details

    • Untriaged
    • Release Note
    • Unknown

    Description

      Install Couchbase Server 5.5.0 on a centos server with IPv6 enable.
      Enable IPv6 on Couchbase Server
      Restart Couchbase Server
      Using raw IPv6 address in browser to initialize this node with service kv, index, n1ql and fts
      Verify IPv6 in Dashboard
      Create a travel sample bucket
      Verify all data and indexes created
      Offline upgrade this node to 5.5.1
      After upgrade, Couchbase Server could not access in UI with raw IPv6 address
      Check config file, IPv6 was disable in Couchbase Server
      Use IPv4 to open Couchbase Server, see Couchbase Server was reset to initial page.

      [root@s10505-ip6 ~]# rpm -i couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm 
      Minimum RAM required  : 4 GB
      System RAM configured : 3.70 GB
       
      Minimum number of processors required : 4 cores
      Number of processors on the system    : 4 cores
       
       
      You have successfully installed Couchbase Server.
      Please browse to http://s10505-ip6:8091/ to configure your server.
      Please refer to http://couchbase.com for additional resources.
       
      Please note that you have to update your firewall configuration to
      allow connections to the following ports:
      4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211,
      11214, 11215, 18091 to 18093, and from 21100 to 21299.
       
      By using this software you agree to the End User License Agreement.
      See /opt/couchbase/LICENSE.txt.
       
      [root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config
      {ipv6, false}.
      [root@s10505-ip6 ~]# systemctl stop couchbase-server
      [root@s10505-ip6 ~]# sed -i '/ipv6, /c \{ipv6, true\}'. /opt/couchbase/etc/couchbase/static_config
      [root@s10505-ip6 ~]# systemctl start couchbase-server
      [root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config
      {ipv6, true}.
      [root@s10505-ip6 ~]# rpm -Uvh couchbase-server-enterprise-5.5.1-centos7.x86_64.rpm 
      Preparing...                          ################################# [100%]
      Minimum RAM required  : 4 GB
      System RAM configured : 3.70 GB
       
      Minimum number of processors required : 4 cores
      Number of processors on the system    : 4 cores
       
      Updating / installing...
         1:couchbase-server-5.5.1-3511      ################################# [ 50%]
      Upgrading couchbase-server ...
        /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes 
      Automatic mode: running without interactive questions or confirmations.
      Analysing...
      Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
      Target node: ns_1@127.0.0.1
      Done: previous node configuration is empty.
      Cleaning up / removing...
         2:couchbase-server-5.5.0-2958      ################################# [100%]
       
      You have successfully installed Couchbase Server.
      Please browse to http://s10505-ip6:8091/ to configure your server.
      Please refer to http://couchbase.com for additional resources.
       
      Please note that you have to update your firewall configuration to
      allow connections to the following ports:
      4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211,
      11214, 11215, 18091 to 18093, and from 21100 to 21299.
       
      By using this software you agree to the End User License Agreement.
      See /opt/couchbase/LICENSE.txt.
       
      [root@s10505-ip6 ~]#
      

      I did try a server with both IPv6 and IPv4 enable or only IPv6 enable. In both cases, I got the same result. Offline upgrade from IPv6 enable server from 5.5.0 to 5.5.x deleted config file which reset node back to initial page.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Bulk closing invalid, duplicate, won't fix bugs. Please feel free to reopen them

            raju Raju Suravarjjala added a comment - Bulk closing invalid, duplicate, won't fix bugs. Please feel free to reopen them
            dfinlay Dave Finlay added a comment -

            This is obviously a bit unfortunate, however, given that 5.5.0 and 5.5.1 are shipped pretty long ago and we have evidence that most folks are on 5.5.2 and later and this only affects single node IPv6, we are planning to document this limitation.

            Ajit to link to doc ticket.

            CC: Eric Schneider, Amarantha Kulkarni

            dfinlay Dave Finlay added a comment - This is obviously a bit unfortunate, however, given that 5.5.0 and 5.5.1 are shipped pretty long ago and we have evidence that most folks are on 5.5.2 and later and this only affects single node IPv6, we are planning to document this limitation. Ajit to link to doc ticket. CC: Eric Schneider , Amarantha Kulkarni

            I uploaded 2 logs from 2 IPv6 nodes cluster.
            fd63.6f75.6368.20d4.423d.37c3.e6f7.3fac-20181010-1021-diag.zip
            fd63.6f75.6368.20d4.9e7.4c7e.6879.d510-20181010-1022-diag.zip

            thuan Thuan Nguyen added a comment - I uploaded 2 logs from 2 IPv6 nodes cluster. fd63.6f75.6368.20d4.423d.37c3.e6f7.3fac-20181010-1021-diag.zip fd63.6f75.6368.20d4.9e7.4c7e.6879.d510-20181010-1022-diag.zip
            dfinlay Dave Finlay added a comment -

            Thuan - can you attach cbcollects for the 2 node case too? Thanks.

            dfinlay Dave Finlay added a comment - Thuan - can you attach cbcollects for the 2 node case too? Thanks.

            So the work around for 2 and more nodes cluster is re-enable IPv6 and restart Couchbase Server after offline upgrade.

            thuan Thuan Nguyen added a comment - So the work around for 2 and more nodes cluster is re-enable IPv6 and restart Couchbase Server after offline upgrade.

            I tried on 2 nodes cluster with IPv6 enable and offline upgrade from 5.5.0 to 5.5.1 (use raw IPv6)
            config.dat file is not deleted but cluster did not up in UI in server has both IPv6 and IPv4 enable or only IPv6 enable. I saw only 2 erlang processes running. After enable IPv6 on all nodes in cluster after upgrade, I could access UI again in both nodes.
            Here are steps I tried.

            • Create a 2 nodes cluster 5.5.0 with IPv6 enable in both node (use raw IPv6).
            • Set services cluster to kv, index, n1ql and fts.
            • Create travel-sample bucket in Settings page.
            • Verify all data loaded to bucket and indexes created in UI.
            • On both nodes, do offline upgrade to 5.5.1
            • Verify upgrade is done in command output.
            • Check UI on both nodes, UI fails to connect.
            • Check erlang processes in nodes, see only 2 processes running.
            • On node with IPv4 enable, check UI with IPv4 address, no connection either (UI down).
            • Check if IPv6 is enable after upgrade, see IPv6 in both nodes are disable.
            • Enable IPv6 in both nodes.
            • Restart Couchbase Server, UI are up on both nodes.
            • Check travel sample if it is still in cluster. Yes, travel sample still in cluster.
            • Check all indexes if they are still in cluster. Yes, all indexes are still in cluster.
            • Check all nodes still retain IPv6 raw address. Yes.

            Here is the output when doing offline upgrade

            [root@s10505-ip6 ~]# rpm -Uhv couchbase-server-enterprise-5.5.1-centos7.x86_64.rpm 
            Preparing...                          ################################# [100%]
            Minimum RAM required  : 4 GB
            System RAM configured : 3.70 GB
             
            Minimum number of processors required : 4 cores
            Number of processors on the system    : 4 cores
             
            Updating / installing...
               1:couchbase-server-5.5.1-3511      ################################# [ 50%]
            Upgrading couchbase-server ...
              /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes 
            Automatic mode: running without interactive questions or confirmations.
            Analysing...
            Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
            Target node: ns_1@FD63:6F75:6368:20D4:423D:37C3:E6F7:3FAC
            Upgrading from 2.0
             
            Couchbase should not be running.
              Please use: /etc/init.d/couchbase-server stop
             
            Database dir: /opt/couchbase/var/lib/couchbase/data
             
            Buckets to upgrade: travel-sample
             
            Checking disk space available for buckets in directory:
              /opt/couchbase/var/lib/couchbase/data
              Free disk bucket space wanted: 0.0
              Free disk bucket space available: 44531814400
              Free disk space factor: 2.0
              Ok.
             
            Analysis complete.
             
            Copying /opt/couchbase/var/lib/couchbase/config/config.dat
                cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
            Copying /opt/couchbase/var/lib/couchbase/ip
                cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip
            Ensuring bucket data directories.
            Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/travel-sample
                mkdir -p /opt/couchbase/var/lib/couchbase/data/travel-sample
            Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
                chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
            Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data
                chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data
             
            Done.
            Cleaning up / removing...
               2:couchbase-server-5.5.0-2958      ################################# [100%]
             
            You have successfully installed Couchbase Server.
            Please browse to http://s10505-ip6:8091/ to configure your server.
            Please refer to http://couchbase.com for additional resources.
             
            Please note that you have to update your firewall configuration to
            allow connections to the following ports:
            4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211,
            11214, 11215, 18091 to 18093, and from 21100 to 21299.
             
            By using this software you agree to the End User License Agreement.
            See /opt/couchbase/LICENSE.txt.
             
            [root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config
            {ipv6, false}.
            [root@s10505-ip6 ~]# ps aux | grep beam.smp
            couchba+ 13536  0.6  0.7 1241736 27308 ?       Ssl  09:38   0:00 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -proto_dist inet_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config"
            couchba+ 14261  104  1.4 1277468 54972 ?       Ssl  09:40   0:01 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -proto_dist inet_tcp -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
            root     14296  0.0  0.0 112660   984 pts/0    S+   09:40   0:00 grep --color=auto beam.smp
            [root@s10505-ip6 ~]# 
            [root@s10505-ip6 ~]# 
            [root@s10505-ip6 ~]# ps aux | grep beam.smp
            couchba+ 13536  0.5  0.6 1241224 27124 ?       Ssl  09:38   0:00 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -proto_dist inet_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config"
            root     14478  0.0  0.0 112660   988 pts/0    S+   09:40   0:00 grep --color=auto beam.smp
            [root@s10505-ip6 ~]# sed -i '/ipv6, /c \{ipv6, true\}'. /opt/couchbase/etc/couchbase/static_config
            [root@s10505-ip6 ~]# systemctl stop couchbase-server
            [root@s10505-ip6 ~]# systemctl start couchbase-server
            [root@s10505-ip6 ~]# ps aux | grep beam.smp
            couchba+ 14768  0.4  0.7 1259476 30860 ?       Ssl  09:41   0:02 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@::1 -proto_dist inet6_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config"
            couchba+ 14866  9.0  3.5 1600656 138800 ?      Ssl  09:41   0:54 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -proto_dist inet6_tcp -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
            couchba+ 14902  0.4  1.2 1035844 49140 ?       Ssl  09:41   0:02 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -pa /opt/couchbase/lib/erlang/lib/appmon-2.1.14.2/ebin /opt/couchbase/lib/erlang/lib/asn1-2.0.4/ebin /opt/couchbase/lib/erlang/lib/compiler-4.9.4/ebin /opt/couchbase/lib/erlang/lib/cosEvent-2.1.14/ebin /opt/couchbase/lib/erlang/lib/cosEventDomain-1.1.13/ebin /opt/couchbase/lib/erlang/lib/cosFileTransfer-1.1.15/ebin /opt/couchbase/lib/erlang/lib/cosNotification-1.1.20/ebin /opt/couchbase/lib/erlang/lib/cosProperty-1.1.16/ebin /opt/couchbase/lib/erlang/lib/cosTime-1.1.13/ebin /opt/couchbase/lib/erlang/lib/cosTransactions-1.2.13/ebin /opt/couchbase/lib/erlang/lib/crypto-3.2/ebin /opt/couchbase/lib/erlang/lib/dialyzer-2.6.1/ebin /opt/couchbase/lib/erlang/lib/diameter-1.5/ebin /opt/couchbase/lib/erlang/lib/edoc-0.7.12.1/ebin /opt/couchbase/lib/erlang/lib/eldap-1.0.2/ebin /opt/couchbase/lib/erlang/lib/erl_docgen-0.3.4.1/ebin /opt/couchbase/lib/erlang/lib/erl_interface-3.7.15 /opt/couchbase/lib/erlang/lib/erts-5.10.4.0.0.1/ebin /opt/couchbase/lib/erlang/lib/et-1.4.4.5/ebin /opt/couchbase/lib/erlang/lib/eunit-2.2.6/ebin /opt/couchbase/lib/erlang/lib/gs-1.5.15.2/ebin /opt/couchbase/lib/erlang/lib/hipe-3.10.2.2/ebin /opt/couchbase/lib/erlang/lib/ic-4.3.4/ebin /opt/couchbase/lib/erlang/lib/inets-5.9.8/ebin /opt/couchbase/lib/erlang/lib/jinterface-1.5.8 /opt/couchbase/lib/erlang/lib/mnesia-4.11/ebin /opt/couchbase/lib/erlang/lib/orber-3.6.26.1/ebin /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/ebin /opt/couchbase/lib/erlang/lib/otp_mibs-1.0.8/ebin /opt/couchbase/lib/erlang/lib/parsetools-2.0.10/ebin /opt/couchbase/lib/erlang/lib/percept-0.8.8.2/ebin /opt/couchbase/lib/erlang/lib/pman-2.7.1.4/ebin /opt/couchbase/lib/erlang/lib/public_key-0.21/ebin /opt/couchbase/lib/erlang/lib/reltool-0.6.4.1/ebin /opt/couchbase/lib/erlang/lib/runtime_tools-1.8.13/ebin /opt/couchbase/lib/erlang/lib/sasl-2.3.4/ebin /opt/couchbase/lib/erlang/lib/snmp-4.25/ebin /opt/couchbase/lib/erlang/lib/ssh-3.0/ebin /opt/couchbase/lib/erlang/lib/ssl-5.3.3/ebin /opt/couchbase/lib/erlang/lib/syntax_tools-1.6.13/ebin /opt/couchbase/lib/erlang/lib/toolbar-1.4.2.3/ebin /opt/couchbase/lib/erlang/lib/tools-2.6.13/ebin /opt/couchbase/lib/erlang/lib/tv-2.1.4.10/ebin /opt/couchbase/lib/erlang/lib/typer-0.9.5/ebin /opt/couchbase/lib/erlang/lib/webtool-0.8.9.2/ebin /opt/couchbase/lib/erlang/lib/xmerl-1.3.6/ebin /opt/couchbase/lib/couchdb/plugins/gc-couchbase-1.0.0/ebin /opt/couchbase/lib/couchdb/plugins/vtree-0.1.0/ebin /opt/couchbase/lib/couchdb/plugins/wkb-1.2.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_dcp-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_index_merger-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_set_view-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/erlang-oauth/ebin /opt/couchbase/lib/couchdb/erlang/lib/etap/ebin /opt/couchbase/lib/couchdb/erlang/lib/lhttpc-1.3/ebin /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/mochiweb-1.4.1/ebin /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/ebin /opt/couchbase/lib/ns_server/erlang/lib/ale/ebin /opt/couchbase/lib/ns_server/erlang/lib/gen_smtp/ebin /opt/couchbase/lib/ns_server/erlang/lib/meck/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_babysitter/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_couchdb/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_server/ebin /opt/couchbase/lib/ns_server/erlang/lib/triq/ebin /opt/couchbase/lib/erlang/lib/stdlib-1.19.4/ebin /opt/couchbase/lib/erlang/lib/kernel-2.16.4/ebin . -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -setcookie xcrfzuwgxrcuedvt -name couchdb_ns_1@::1 -smp enable -kernel error_logger false -sasl sasl_error_logger false -nouser -hidden -proto_dist inet6_tcp -run child_erlang child_start ns_couchdb
            root     15196  0.0  0.0 112664   988 pts/0    S+   09:51   0:00 grep --color=auto beam.smp
            [root@s10505-ip6 ~]# 
            

            I will collect logs and upload here soon

            thuan Thuan Nguyen added a comment - I tried on 2 nodes cluster with IPv6 enable and offline upgrade from 5.5.0 to 5.5.1 (use raw IPv6) config.dat file is not deleted but cluster did not up in UI in server has both IPv6 and IPv4 enable or only IPv6 enable. I saw only 2 erlang processes running. After enable IPv6 on all nodes in cluster after upgrade, I could access UI again in both nodes. Here are steps I tried. Create a 2 nodes cluster 5.5.0 with IPv6 enable in both node (use raw IPv6). Set services cluster to kv, index, n1ql and fts. Create travel-sample bucket in Settings page. Verify all data loaded to bucket and indexes created in UI. On both nodes, do offline upgrade to 5.5.1 Verify upgrade is done in command output. Check UI on both nodes, UI fails to connect. Check erlang processes in nodes, see only 2 processes running. On node with IPv4 enable, check UI with IPv4 address, no connection either (UI down). Check if IPv6 is enable after upgrade, see IPv6 in both nodes are disable. Enable IPv6 in both nodes. Restart Couchbase Server, UI are up on both nodes. Check travel sample if it is still in cluster. Yes, travel sample still in cluster. Check all indexes if they are still in cluster. Yes, all indexes are still in cluster. Check all nodes still retain IPv6 raw address. Yes. Here is the output when doing offline upgrade [root@s10505-ip6 ~]# rpm -Uhv couchbase-server-enterprise-5.5.1-centos7.x86_64.rpm Preparing... ################################# [100%] Minimum RAM required : 4 GB System RAM configured : 3.70 GB   Minimum number of processors required : 4 cores Number of processors on the system : 4 cores   Updating / installing... 1:couchbase-server-5.5.1-3511 ################################# [ 50%] Upgrading couchbase-server ... /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat Target node: ns_1@FD63:6F75:6368:20D4:423D:37C3:E6F7:3FAC Upgrading from 2.0   Couchbase should not be running. Please use: /etc/init.d/couchbase-server stop   Database dir: /opt/couchbase/var/lib/couchbase/data   Buckets to upgrade: travel-sample   Checking disk space available for buckets in directory: /opt/couchbase/var/lib/couchbase/data Free disk bucket space wanted: 0.0 Free disk bucket space available: 44531814400 Free disk space factor: 2.0 Ok.   Analysis complete.   Copying /opt/couchbase/var/lib/couchbase/config/config.dat cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Copying /opt/couchbase/var/lib/couchbase/ip cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/travel-sample mkdir -p /opt/couchbase/var/lib/couchbase/data/travel-sample Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data   Done. Cleaning up / removing... 2:couchbase-server-5.5.0-2958 ################################# [100%]   You have successfully installed Couchbase Server. Please browse to http://s10505-ip6:8091/ to configure your server. Please refer to http://couchbase.com for additional resources.   Please note that you have to update your firewall configuration to allow connections to the following ports: 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 11214, 11215, 18091 to 18093, and from 21100 to 21299.   By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt.   [root@s10505-ip6 ~]# grep ipv6 /opt/couchbase/etc/couchbase/static_config {ipv6, false}. [root@s10505-ip6 ~]# ps aux | grep beam.smp couchba+ 13536 0.6 0.7 1241736 27308 ? Ssl 09:38 0:00 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -proto_dist inet_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" couchba+ 14261 104 1.4 1277468 54972 ? Ssl 09:40 0:01 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -proto_dist inet_tcp -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini root 14296 0.0 0.0 112660 984 pts/0 S+ 09:40 0:00 grep --color=auto beam.smp [root@s10505-ip6 ~]# [root@s10505-ip6 ~]# [root@s10505-ip6 ~]# ps aux | grep beam.smp couchba+ 13536 0.5 0.6 1241224 27124 ? Ssl 09:38 0:00 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -proto_dist inet_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" root 14478 0.0 0.0 112660 988 pts/0 S+ 09:40 0:00 grep --color=auto beam.smp [root@s10505-ip6 ~]# sed -i '/ipv6, /c \{ipv6, true\}'. /opt/couchbase/etc/couchbase/static_config [root@s10505-ip6 ~]# systemctl stop couchbase-server [root@s10505-ip6 ~]# systemctl start couchbase-server [root@s10505-ip6 ~]# ps aux | grep beam.smp couchba+ 14768 0.4 0.7 1259476 30860 ? Ssl 09:41 0:02 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false global_enable_tracing true -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@::1 -proto_dist inet6_tcp -setcookie nocookie -- -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" couchba+ 14866 9.0 3.5 1600656 138800 ? Ssl 09:41 0:54 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -proto_dist inet6_tcp -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini couchba+ 14902 0.4 1.2 1035844 49140 ? Ssl 09:41 0:02 /opt/couchbase/lib/erlang/erts-5.10.4.0.0.1/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname erl -- -home /home/couchbase -- -pa /opt/couchbase/lib/erlang/lib/appmon-2.1.14.2/ebin /opt/couchbase/lib/erlang/lib/asn1-2.0.4/ebin /opt/couchbase/lib/erlang/lib/compiler-4.9.4/ebin /opt/couchbase/lib/erlang/lib/cosEvent-2.1.14/ebin /opt/couchbase/lib/erlang/lib/cosEventDomain-1.1.13/ebin /opt/couchbase/lib/erlang/lib/cosFileTransfer-1.1.15/ebin /opt/couchbase/lib/erlang/lib/cosNotification-1.1.20/ebin /opt/couchbase/lib/erlang/lib/cosProperty-1.1.16/ebin /opt/couchbase/lib/erlang/lib/cosTime-1.1.13/ebin /opt/couchbase/lib/erlang/lib/cosTransactions-1.2.13/ebin /opt/couchbase/lib/erlang/lib/crypto-3.2/ebin /opt/couchbase/lib/erlang/lib/dialyzer-2.6.1/ebin /opt/couchbase/lib/erlang/lib/diameter-1.5/ebin /opt/couchbase/lib/erlang/lib/edoc-0.7.12.1/ebin /opt/couchbase/lib/erlang/lib/eldap-1.0.2/ebin /opt/couchbase/lib/erlang/lib/erl_docgen-0.3.4.1/ebin /opt/couchbase/lib/erlang/lib/erl_interface-3.7.15 /opt/couchbase/lib/erlang/lib/erts-5.10.4.0.0.1/ebin /opt/couchbase/lib/erlang/lib/et-1.4.4.5/ebin /opt/couchbase/lib/erlang/lib/eunit-2.2.6/ebin /opt/couchbase/lib/erlang/lib/gs-1.5.15.2/ebin /opt/couchbase/lib/erlang/lib/hipe-3.10.2.2/ebin /opt/couchbase/lib/erlang/lib/ic-4.3.4/ebin /opt/couchbase/lib/erlang/lib/inets-5.9.8/ebin /opt/couchbase/lib/erlang/lib/jinterface-1.5.8 /opt/couchbase/lib/erlang/lib/mnesia-4.11/ebin /opt/couchbase/lib/erlang/lib/orber-3.6.26.1/ebin /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/ebin /opt/couchbase/lib/erlang/lib/otp_mibs-1.0.8/ebin /opt/couchbase/lib/erlang/lib/parsetools-2.0.10/ebin /opt/couchbase/lib/erlang/lib/percept-0.8.8.2/ebin /opt/couchbase/lib/erlang/lib/pman-2.7.1.4/ebin /opt/couchbase/lib/erlang/lib/public_key-0.21/ebin /opt/couchbase/lib/erlang/lib/reltool-0.6.4.1/ebin /opt/couchbase/lib/erlang/lib/runtime_tools-1.8.13/ebin /opt/couchbase/lib/erlang/lib/sasl-2.3.4/ebin /opt/couchbase/lib/erlang/lib/snmp-4.25/ebin /opt/couchbase/lib/erlang/lib/ssh-3.0/ebin /opt/couchbase/lib/erlang/lib/ssl-5.3.3/ebin /opt/couchbase/lib/erlang/lib/syntax_tools-1.6.13/ebin /opt/couchbase/lib/erlang/lib/toolbar-1.4.2.3/ebin /opt/couchbase/lib/erlang/lib/tools-2.6.13/ebin /opt/couchbase/lib/erlang/lib/tv-2.1.4.10/ebin /opt/couchbase/lib/erlang/lib/typer-0.9.5/ebin /opt/couchbase/lib/erlang/lib/webtool-0.8.9.2/ebin /opt/couchbase/lib/erlang/lib/xmerl-1.3.6/ebin /opt/couchbase/lib/couchdb/plugins/gc-couchbase-1.0.0/ebin /opt/couchbase/lib/couchdb/plugins/vtree-0.1.0/ebin /opt/couchbase/lib/couchdb/plugins/wkb-1.2.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_dcp-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_index_merger-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_set_view-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/erlang-oauth/ebin /opt/couchbase/lib/couchdb/erlang/lib/etap/ebin /opt/couchbase/lib/couchdb/erlang/lib/lhttpc-1.3/ebin /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/mochiweb-1.4.1/ebin /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/ebin /opt/couchbase/lib/ns_server/erlang/lib/ale/ebin /opt/couchbase/lib/ns_server/erlang/lib/gen_smtp/ebin /opt/couchbase/lib/ns_server/erlang/lib/meck/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_babysitter/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_couchdb/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_server/ebin /opt/couchbase/lib/ns_server/erlang/lib/triq/ebin /opt/couchbase/lib/erlang/lib/stdlib-1.19.4/ebin /opt/couchbase/lib/erlang/lib/kernel-2.16.4/ebin . -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -setcookie xcrfzuwgxrcuedvt -name couchdb_ns_1@::1 -smp enable -kernel error_logger false -sasl sasl_error_logger false -nouser -hidden -proto_dist inet6_tcp -run child_erlang child_start ns_couchdb root 15196 0.0 0.0 112664 988 pts/0 S+ 09:51 0:00 grep --color=auto beam.smp [root@s10505-ip6 ~]# I will collect logs and upload here soon

            I will try 2 IPv6 nodes cluster tomorrow if I could reproduce this issue.

            thuan Thuan Nguyen added a comment - I will try 2 IPv6 nodes cluster tomorrow if I could reproduce this issue.

            People

              ajit.yagaty Ajit Yagaty [X] (Inactive)
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty