Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7238

ns_server is still validating ip address in ip file even if erlang already has node name defined (was: 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one.)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta-2, 2.0
    • Fix Version/s: 2.0
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None

      Description

      Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

      Error messages from the log:
      [ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
      [ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

      The logs are available in the link below:
      https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip

      update:

      Apparently as part of process of setting up node name folks just left original /opt/couchbase/var/lib/couchbase/ip. And ns_server's bug is due to attempt to validate that address even though it won't be actually used.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        balak balak created issue -
        balak balak made changes -
        Field Original Value New Value
        Assignee Abhinav Dangeti [ abhinav ]
        ketaki Ketaki Gangal made changes -
        Fix Version/s 2.0 [ 10114 ]
        Affects Version/s 2.0 [ 10114 ]
        abhinav Abhinav Dangeti made changes -
        Assignee Abhinav Dangeti [ abhinav ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        steve Steve Yen added a comment -

        abhinav now attempting to reproduce

        Show
        steve Steve Yen added a comment - abhinav now attempting to reproduce
        steve Steve Yen made changes -
        Summary 2.0 Bulid 1941 : Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one.
        Assignee Aleksey Kondratenko [ alkondratenko ] Abhinav Dangeti [ abhinav ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        steve Steve Yen added a comment -

        please also get DNS diagnostic info... like ping (ask alk).

        it could be as simple as DNS propagation delay.

        Show
        steve Steve Yen added a comment - please also get DNS diagnostic info... like ping (ask alk). it could be as simple as DNS propagation delay.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        From error message it appears that DNS resolver still thinks old ip is assigned to this hostname.

        So in order to help diagnosing this I need both cbcollect_info (or just output of ifconfig -a) and some information about in what ip this hostname is resolved. Simple way is by pinging hostname and sending me output

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - From error message it appears that DNS resolver still thinks old ip is assigned to this hostname. So in order to help diagnosing this I need both cbcollect_info (or just output of ifconfig -a) and some information about in what ip this hostname is resolved. Simple way is by pinging hostname and sending me output
        Hide
        abhinav Abhinav Dangeti added a comment -
        • So started with 10.1.3.235, 10.1.3.236 (build 1954)
        • set host ip's on /etc/hosts
        • stopped couchbase-server on 10.1.3.236
        • changed ip of 10.1.3.236 to 10.1.3.222
        • resolved /etc/hosts to the new ip
        • started couchbase-server back up on 10.1.3.222, server never comes back up.

        [ Servers available as is ]

        10.1.3.222>>
        ifconfig -a
        eth0 Link encap:Ethernet HWaddr 00:50:56:97:02:D2
        inet addr:10.1.3.222 Bcast:10.255.255.255 Mask:255.0.0.0
        inet6 addr: fe80::250:56ff:fe97:2d2/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:109556236 errors:0 dropped:0 overruns:0 frame:0
        TX packets:108558164 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:83579290130 (77.8 GiB) TX bytes:86321632329 (80.3 GiB)

        lo Link encap:Local Loopback
        inet addr:127.0.0.1 Mask:255.0.0.0
        inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING MTU:16436 Metric:1
        RX packets:72816990 errors:0 dropped:0 overruns:0 frame:0
        TX packets:72816990 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:120374350544 (112.1 GiB) TX bytes:120374350544 (112.1 GiB)

        sit0 Link encap:IPv6-in-IPv4
        NOARP MTU:1480 Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

        <<Attaching cbcollectinfo_10_1_3_222.zip>>

        Show
        abhinav Abhinav Dangeti added a comment - So started with 10.1.3.235, 10.1.3.236 (build 1954) set host ip's on /etc/hosts stopped couchbase-server on 10.1.3.236 changed ip of 10.1.3.236 to 10.1.3.222 resolved /etc/hosts to the new ip started couchbase-server back up on 10.1.3.222, server never comes back up. [ Servers available as is ] 10.1.3.222>> ifconfig -a eth0 Link encap:Ethernet HWaddr 00:50:56:97:02:D2 inet addr:10.1.3.222 Bcast:10.255.255.255 Mask:255.0.0.0 inet6 addr: fe80::250:56ff:fe97:2d2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:109556236 errors:0 dropped:0 overruns:0 frame:0 TX packets:108558164 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:83579290130 (77.8 GiB) TX bytes:86321632329 (80.3 GiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:72816990 errors:0 dropped:0 overruns:0 frame:0 TX packets:72816990 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:120374350544 (112.1 GiB) TX bytes:120374350544 (112.1 GiB) sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) <<Attaching cbcollectinfo_10_1_3_222.zip>>
        abhinav Abhinav Dangeti made changes -
        Attachment cbcollectinfo_10_1_3_222.zip [ 15859 ]
        abhinav Abhinav Dangeti made changes -
        Assignee Abhinav Dangeti [ abhinav ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        a) cannot access .222.

        b) don't have ping output that I need in order to understand more

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - a) cannot access .222. b) don't have ping output that I need in order to understand more
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Abhinav Dangeti [ abhinav ]
        james.mauss James Mauss made changes -
        Description
        Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

        Error messages from the log:
        [ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
        [ns_server:warn,2012-11-16T13:16:45.522,ns_1@couch14.dev.jawfishgames.com:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

        The logs are available in the link below:
        https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip







        Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

        Error messages from the log:
        [ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
        [ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

        The logs are available in the link below:
        https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip






        Hide
        abhinav Abhinav Dangeti added a comment - - edited

        If we use ifconfig eth0 10.1.3.222, to change the IP, we see the issue.
        However the issue doesn't occur when the IP is changed this way:
        vim /etc/sysconfig/network-scripts/ifcfg-eth0

        (and comment out the BOOTPROTO=dhcp and set it to static)
        ..

        1. Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
          DEVICE=eth0
          #BOOTPROTO=dhcp
          BOOTPROTO=static
          ONBOOT=yes
          IPADDR=10.1.3.222
          GATEWAY=10.1.0.1
          NETMASK=255.255.0.0
          ...
          sudo /etc/init.d/network restart

        The reason why this worked was because /opt/couchbase/var/lib/couchbase/ip was empty.

        Show
        abhinav Abhinav Dangeti added a comment - - edited If we use ifconfig eth0 10.1.3.222, to change the IP, we see the issue. However the issue doesn't occur when the IP is changed this way: vim /etc/sysconfig/network-scripts/ifcfg-eth0 (and comment out the BOOTPROTO=dhcp and set it to static) .. Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) DEVICE=eth0 #BOOTPROTO=dhcp BOOTPROTO=static ONBOOT=yes IPADDR=10.1.3.222 GATEWAY=10.1.0.1 NETMASK=255.255.0.0 ... sudo /etc/init.d/network restart The reason why this worked was because /opt/couchbase/var/lib/couchbase/ip was empty.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Looks like /opt/couchbase/var/lib/.../ip is still being used somehow. I recommend manually deleting it. It's still ns_server's bug if we try do anything about it when hostname is specificied

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Looks like /opt/couchbase/var/lib/.../ip is still being used somehow. I recommend manually deleting it. It's still ns_server's bug if we try do anything about it when hostname is specificied
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Summary 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. ns_server is still validating ip address in ip file even if erlang already has node name defined (was: 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one.)
        Description
        Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

        Error messages from the log:
        [ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
        [ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

        The logs are available in the link below:
        https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip







        Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

        Error messages from the log:
        [ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
        [ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

        The logs are available in the link below:
        https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip

        update:

        Apparently as part of process of setting up node name folks just left original /opt/couchbase/var/lib/couchbase/ip. And ns_server's bug is due to attempt to validate that address even though it won't be actually used.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Abhinav Dangeti [ abhinav ] Aleksey Kondratenko [ alkondratenko ]
        steve Steve Yen made changes -
        Component/s ns_server [ 10019 ]
        Hide
        steve Steve Yen added a comment -

        The ip file was/is being used by cbupgrade situation. Please see...

        http://www.couchbase.com/issues/browse/MB-7241

        Show
        steve Steve Yen added a comment - The ip file was/is being used by cbupgrade situation. Please see... http://www.couchbase.com/issues/browse/MB-7241
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Approved for 2.0. Be careful with using right branch

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Approved for 2.0. Be careful with using right branch
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Aliaksey Artamonau [ aliaksey artamonau ]
        Hide
        steve Steve Yen added a comment - - edited

        i think this fix was merged? – http://review.couchbase.org/#/c/22895/

        Show
        steve Steve Yen added a comment - - edited i think this fix was merged? – http://review.couchbase.org/#/c/22895/
        Aliaksey Artamonau Aliaksey Artamonau made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Aliaksey Artamonau Aliaksey Artamonau
            Reporter:
            balak balak
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes