Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.6.0
-
Initial server version - 7.1.2-3454
Upgrade version - 7.6.0-2032
OS - Debian 12
-
Untriaged
-
Linux x86_64
-
-
0
-
Yes
Description
Steps:
- Install version 7.1.2-3454 on 3 nodes and initialise a cluster with these 3 nodes running just the KV service.
- Create a Magma bucket with 2 replicas.
- Load 100 million documents into the bucket. Bucket is at 1% DGM/resident ratio.
- Start the process of offline upgrade to 7.6.0-2032.
- Stop Couchbase service on all nodes, uninstall the previous version and install the version 7.6.0-2032 on all the nodes.
- That completes the process of upgrade to 7.6
- Enable CDC on the Magma bucket, create new collections with history true and false.
- Create a new Magma bucket and load 10,000 docs into this bucket.
- Rebalance-in a new node (172.23.217.172) running 7.6.0-2032, rebalance completes successfully.
- Rebalance-out the same node (172.23.217.172) which was brought in, in the previous step. Rebalance fails this time with the reason,
{mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.32454.14>, |
{socket_closed,
|
{gen_server,call,
|
[{'janitor_agent-bucket-0', |
'ns_1@172.23.217.169'}, |
{if_rebalance,<0.13535.13>, |
{wait_dcp_data_move,
|
['ns_1@172.23.217.171', |
'ns_1@172.23.217.170'], |
329}}, |
infinity]}}}}}.
|
Rebalance Operation Id = 7b0592359b604e90847c12218a9e4aaf
|
- Cb-collect logs have been attached.
From node 172.23.217.169 in the ns_server.info.log we see
[ns_server:info,2024-01-22T00:20:55.557-08:00,ns_1@172.23.217.169:rebalance_agent<0.987.0>:rebalance_agent:handle_down:290]Rebalancer process <22316.13001.13> died (reason {mover_crashed, |
{unexpected_exit,
|
{'EXIT',<22316.32454.14>, |
{socket_closed,
|
{gen_server,call,
|
[{'janitor_agent-bucket-0', |
'ns_1@172.23.217.169'}, |
{if_rebalance,
|
<22316.13535.13>, |
{wait_dcp_data_move,
|
['ns_1@172.23.217.171', |
'ns_1@172.23.217.170'], |
329}}, |
infinity]}}}}}).
|
On node 172.23.217.170 in memcached.log we see the disconnect
2024-01-22T00:20:55.514800-08:00 INFO 13085: (No Engine) DCP (Producer) eq_dcpq:replication:ns_1@172.23.217.170->ns_1@172.23.217.169:bucket-0 - Removing connection [ {"ip":"172.23.217.169","port":45336} - {"ip":"172.23.217.170","port":11206} (System, @ns_server) ]
|
On node 172.23.217.171 in memcached.log we see the disconnect
2024-01-22T00:20:55.762402-08:00 INFO 67273: (No Engine) DCP (Producer) eq_dcpq:replication:ns_1@172.23.217.171->ns_1@172.23.217.170:bucket-0 - Removing connection [ {"ip":"172.23.217.170","port":45636}
|
- {"ip":"172.23.217.171","port":11206} (System, @ns_server) ]
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-