Critical Description
When enabling ipv6 in couchbase it uses different ports.
Here is ipv4:
$ sudo -u app lsof -P -p 2506 2> /dev/null | grep LISTEN
|
beam.smp 2506 app 25u IPv4 57804549 0t0 TCP *:21101 (LISTEN)
|
beam.smp 2506 app 29u IPv4 57837941 0t0 TCP *:18091 (LISTEN)
|
beam.smp 2506 app 40u IPv4 57810008 0t0 TCP *:8091 (LISTEN)
|
In this case 8091, 18091, and 21101 are all documented here: https://docs.couchbase.com/server/5.5/install/install-ports.html
Here's ipv6:
$ sudo -u app lsof -P -p 3305 2> /dev/null | grep LISTEN
|
beam.smp 3305 app 25u IPv6 77218341 0t0 TCP *:35211 (LISTEN)
|
beam.smp 3305 app 36u IPv6 77220042 0t0 TCP *:8091 (LISTEN)
|
beam.smp 3305 app 42u IPv6 77218407 0t0 TCP *:18091 (LISTEN)
|
So what used to be listening on port 21101 is now listening on 35211. This port varies from node to node but is always outside of any known port range listed in any couchbase documentation.
This is a blocker for us to roll out ipv6 as we have firewall rules in place between our couchbase nodes and need to explicitly whitelist all ports to open up. With couchbase listening on dynamic ports then we can start up a cluster on nodeA but we can't join nodeB to it due to this (from debug.log where /controller/addNode was called):
|
|
[cluster:debug,2018-09-20T14:32:47.038Z,ns_1@lor1-0000043.int.linkedin.com:ns_cluster<0.152.0>:ns_cluster:verify_otp_connectivity:626]port_please("ns_1", "lor1-0000264.int.linkedin.com") = 39518
|
[cluster:debug,2018-09-20T14:32:47.038Z,ns_1@lor1-0000043.int.linkedin.com:ns_cluster<0.152.0>:ns_cluster:handle_call:176]add_node("lor1-0000264.int.linkedin.com", 8091, undefined, ..) -> {error,
|
connect_node,
|
<<"Failed to reach otp port 39518 for node [\" \",\n <<\"Could not connect to \\\"lor1-0000264.int.linkedin.com\\\" on port \\\"39518\\\". This could be due to an incorrect host/port combination or a firewall in place between the servers.\">>].ns_1@lor1-0000264.int.linkedin.com This can be firewall problem.">>,
|
{error,
|
econnrefused}}
|
[user:info,2018-09-20T14:32:47.038Z,ns_1@lor1-0000043.int.linkedin.com:<0.1920.0>:ns_cluster:add_node_to_group:82]Failed to add node lor1-0000264.int.linkedin.com:8091 to cluster. Failed to reach otp port 39518 for node [" ",
|
<<"Could not connect to \"lor1-0000264.int.linkedin.com\" on port \"39518\". This could be due to an incorrect host/port combination or a firewall in place between the servers.">>].ns_1@lor1-0000264.int.linkedin.com This can be firewall problem.
|
To reproduce this:
- install couchbase
- enable ipv6 as documented at https://developer.couchbase.com/documentation/server/current/install/ipv6-setup.html
- start up couchbase
- observe LISTEN tcp ports on the beam.smp process that is listening on 8091 and 18091