Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.6.0
-
Operating System : Debian GNU/Linux 10 (buster)
Initial Version : Couchbase Enterprise Edition 7.2.0-5325
Upgrade Version : Couchbase Enterprise Edition 7.6.0-2005
-
Untriaged
-
Linux x86_64
-
-
0
-
Unknown
Description
Steps to reproduce
- Created a 2 node kv cluster with Couchbase Enterprise Edition 7.2.0-5325 172.23.121.183 and 172.23.121.189
- Created a couchstore bucket named "standard_bucket0" and loaded a few items onto it
- Installed Couchbase Enterprise Edition 7.6.0-2005 on 172.23.121.66 and 172.23.121.144
- Swap rebalanced the existing nodes with the new nodes
Rebalance fails
2024-01-15T19:26:27.936-08:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.121.144) - Worker <0.31134.0> (for action {move,{875, ['ns_1@172.23.121.189', 'ns_1@172.23.121.183'], ['ns_1@172.23.121.66', 'ns_1@172.23.121.144'], []}}) exited with reason {unexpected_exit, {'EXIT', <0.31588.0>, {{{{{badmatch, [{<35616.13736.0>, {done, exit, {{{badmatch, {error, etimedout}}, [{dcp_proxy, connect_inner, 3, [{file, "src/dcp_proxy.erl"}, {line, 299}]}, {dcp_proxy, connect, 5, [{file, "src/dcp_proxy.erl"}, {line, 252}]}, {dcp_proxy, maybe_connect, 2, [{file, "src/dcp_proxy.erl"}, {line, 235}]}, {dcp_producer_conn, handle_call, 4, [{file, "src/dcp_producer_conn.erl"}, {line, 50}]}, {dcp_proxy, handle_call, 3, [{file, "src/dcp_proxy.erl"}, {line, 154}]}, {gen_server, try_handle_call, 4, [{file, "gen_server.erl"}, {line, 1149}]}, {gen_server, handle_msg, 6, [{file, "gen_server.erl"}, {line, 1178}]}, {proc_lib, init_p_do_apply, 3, [{file, "proc_lib.erl"}, {line, 240}]}]}, {gen_server, call, [<35616.13735.0>, {connect, [collections, del_times, del_user_xattr, json, set_consumer_name, snappy, xattr]}, infinity]}}, [{gen_server, call, 3, [{file, "gen_server.erl"}, {line, 385}]}, {dcp_replicator, connect_to_producer, 3, [{file, "src/dcp_replicator.erl"}, {line, 76}]}, {dcp_replicator, '-spawn_and_wait/1-fun-0-', 1, [{file, "src/dcp_replicator.erl"}, {line, 323}]}]}}]}, [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file, "src/misc.erl"}, {line, 1517}]}, {dcp_replicator, spawn_and_wait, 1, [{file, "src/dcp_replicator.erl"}, {line, 344}]}, {dcp_replicator, handle_info, 2, [{file, "src/dcp_replicator.erl"}, {line, 137}]}, {gen_server, try_dispatch, 4, [{file, "gen_server.erl"}, {line, 1123}]}, {gen_server, handle_msg, 6, [{file, "gen_server.erl"}, {line, 1200}]}, {proc_lib, init_p_do_apply, 3, [{file, "proc_lib.erl"}, {line, 240}]}]}, {gen_server, call, [<35616.13733.0>, get_partitions, infinity]}}, {gen_server, call, ['dcp_replication_manager-standard_bucket0', {get_replicator_pid, 358}, infinity]}}, {gen_server, call, [{'janitor_agent-standard_bucket0', 'ns_1@172.23.121.66'}, {if_rebalance, <0.7446.0>, {update_vbucket_state, 875, active, undefined, undefined, [['ns_1@172.23.121.66', 'ns_1@172.23.121.144']]}}, infinity]}}}}2024-01-15T19:26:27.977-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.121.144) - Rebalance exited with reason {mover_crashed, {unexpected_exit, {'EXIT',<0.31588.0>, {{{{{badmatch, [{<35616.13736.0>, {done,exit, {{{badmatch,{error,etimedout}}, [{dcp_proxy,connect_inner,3, [{file,"src/dcp_proxy.erl"}, {line,299}]}, {dcp_proxy,connect,5, [{file,"src/dcp_proxy.erl"}, {line,252}]}, {dcp_proxy,maybe_connect,2, [{file,"src/dcp_proxy.erl"}, {line,235}]}, {dcp_producer_conn,handle_call,4, [{file, "src/dcp_producer_conn.erl"}, {line,50}]}, {dcp_proxy,handle_call,3, [{file,"src/dcp_proxy.erl"}, {line,154}]}, {gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,1149}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,1178}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,240}]}]}, {gen_server,call, [<35616.13735.0>, {connect, [collections,del_times, del_user_xattr,json, set_consumer_name,snappy,xattr]}, infinity]}}, [{gen_server,call,3, [{file,"gen_server.erl"}, {line,385}]}, {dcp_replicator,connect_to_producer, 3, [{file,"src/dcp_replicator.erl"}, {line,76}]}, {dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file,"src/dcp_replicator.erl"}, {line,323}]}]}}]}, [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"},{line,1517}]}, {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,344}]}, {dcp_replicator,handle_info,2, [{file,"src/dcp_replicator.erl"}, {line,137}]}, {gen_server,try_dispatch,4, [{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,240}]}]}, {gen_server,call, [<35616.13733.0>,get_partitions, infinity]}}, {gen_server,call, ['dcp_replication_manager-standard_bucket0', {get_replicator_pid,358}, infinity]}}, {gen_server,call, [{'janitor_agent-standard_bucket0', 'ns_1@172.23.121.66'}, {if_rebalance,<0.7446.0>, {update_vbucket_state,875,active, undefined,undefined, [['ns_1@172.23.121.66', 'ns_1@172.23.121.144']]}}, infinity]}}}}}.Rebalance Operation Id = 4b7d5448ac593ddd391ca4a66073f753 |
Testrunner script to reproduce
./testrunner -i /data/workspace/debian-p0-upgrade-vset00-00-old-from-72x/testexec.60533.ini -p check_seqno=false,upgrade_version=7.6.0-2005,stop-on-failure=False,skip_cleanup=False,skip_init_check_cbserver=true,GROUP=P0,skip_log_scan=true,upgrade_test=true,get-cbcollect-info=True,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 -t newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_out,initial_version=7.2.0-5325,default_bucket=False,sasl_buckets=1,standard_buckets=1,items=10000,GROUP=3_0;ONLINE;WINDOWS;P0 |
Job name : debian-upgrade-old-from-72x
Attachments
Issue Links
- duplicates
-
-
- Closed
-