Couchnode 2.3.2/libcouchbase 2.7.4
Couchbase 4.5.1 Cluster
Before swap/rebalance Couchbase cluster consists of nodes 101 and 105
After swap/rebalance Couchbase cluster consists of nodes 101 and 102
Couchnode 2.3.2 client is running N1QL queries again Couchbase 4.5.1, a two node cluster.
A swap rebalance is done where a node is removed (105), and another is added (102).
Right as the rebalance is finished, a query is done where the index being used was on the node 105.
The connection to 8093 on node 105 fails and causes a cluster map refresh.
A 'Hello Request' is sent to both node 101 and 105 on port 11210.
Both node 105 and 101 respond.
Node 105 responds first and both go through the SASL Auth process.
At this point no more requests are sent to node 101.
Node 105 I assume has shutdown or stopped replying on port 11210 at this point.
Couchnode (or libcouchbase) continues to keep trying node 105 over and over and a new cluster map is never downloaded.
I have included the tcpdump output from this transaction. The first connection reset to node 105 on port 8093 happens at 14:45:42.557965. From there you can follow the events.