Details
-
Bug
-
Resolution: Duplicate
-
Major
-
7.2.5
-
Operating System : Amazon Linux 2023
Couchbase Enterprise Edition 7.2.5-7595
-
Untriaged
-
-
0
-
Unknown
Description
Steps to reproduce
- Created a 4 node cluster with following services
- ec2-50-17-111-133.compute-1.amazonaws.com - kv
- ec2-54-234-55-192.compute-1.amazonaws.com - kv
- ec2-54-242-38-198.compute-1.amazonaws.com - kv
- ec2-54-196-176-35.compute-1.amazonaws.com - kv, index, n1ql
- Created a couchstore bucket named bucket-12 with 2 replicas
- Loaded 1000000 documents onto it
- Started a swap rebalance by adding 2 kv nodes and removing 2 kv nodes
- During rebalance started a transaction using cbq
- Ran multiple queries
- Rebalance fails
2024-04-15T04:02:01.871Z, ns_vbucket_mover:0:critical:message(ns_1@ec2-54-196-176-35.compute-1.amazonaws.com) - Worker <0.14864.205> (for action {move, {979, ['ns_1@ec2-54-242-38-198.compute-1.amazonaws.com', 'ns_1@ec2-54-234-55-192.compute-1.amazonaws.com', 'ns_1@ec2-50-17-111-133.compute-1.amazonaws.com'], ['ns_1@ec2-54-242-31-146.compute-1.amazonaws.com', 'ns_1@ec2-54-234-55-192.compute-1.amazonaws.com', 'ns_1@ec2-34-203-199-75.compute-1.amazonaws.com'], []}}) exited with reason {unexpected_exit, {'EXIT', <0.15283.205>, {{wait_seqno_persisted_failed, "bucket-12", 979,1203, [{'ns_1@ec2-34-203-199-75.compute-1.amazonaws.com', {'EXIT', {{{{{badmatch, [{<34927.31444.58>, {done, exit, {badarg, {gen_server, call, [<34927.31461.58>, {connect, [collections, del_times, del_user_xattr, json, set_consumer_name, snappy, xattr]}, infinity]}}, [{gen_server, call, 3, [{file, "gen_server.erl"}, {line, 385}]}, {dcp_replicator, connect_to_producer, 3, [{file, "src/dcp_replicator.erl"}, {line, 76}]}, {dcp_replicator, '-spawn_and_wait/1-fun-0-', 1, [{file, "src/dcp_replicator.erl"}, {line, 336}]}]}}]}, [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file, "src/misc.erl"}, {line, 1470}]}, {dcp_replicator, spawn_and_wait, 1, [{file, "src/dcp_replicator.erl"}, {line, 357}]}, {dcp_replicator, handle_info, 2, [{file, "src/dcp_replicator.erl"}, {line, 137}]}, {gen_server, try_dispatch, 4, [{file, "gen_server.erl"}, {line, 1123}]}, {gen_server, handle_msg, 6, [{file, "gen_server.erl"}, {line, 1200}]}, {proc_lib, init_p_do_apply, 3, [{file, "proc_lib.erl"}, {line, 240}]}]}, {gen_server, call, [<34927.31455.58>, get_partitions, infinity]}}, {gen_server, call, ['dcp_replication_manager-bucket-12', {get_replicator_pid, 250}, infinity]}}, {gen_server, call, [{'janitor_agent-bucket-12', 'ns_1@ec2-34-203-199-75.compute-1.amazonaws.com'}, {if_rebalance, <0.10696.204>, {wait_seqno_persisted, 979, 1203}}, infinity]}}}}]}, [{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-', 5, [{file, "src/ns_single_vbucket_mover.erl"}, {line, 474}]}, {proc_lib, init_p,3, [{file, "proc_lib.erl"}, {line, 225}]}]}}}2024-04-15T04:02:01.981Z, ns_orchestrator:0:critical:message(ns_1@ec2-54-196-176-35.compute-1.amazonaws.com) - Rebalance exited with reason {mover_crashed, {unexpected_exit, {'EXIT',<0.15283.205>, {{wait_seqno_persisted_failed,"bucket-12", 979,1203, [{'ns_1@ec2-34-203-199-75.compute-1.amazonaws.com', {'EXIT', {{{{{badmatch, [{<34927.31444.58>, {done,exit, {badarg, {gen_server,call, [<34927.31461.58>, {connect, [collections,del_times, del_user_xattr,json, set_consumer_name,snappy, xattr]}, infinity]}}, [{gen_server,call,3, [{file,"gen_server.erl"}, {line,385}]}, {dcp_replicator, connect_to_producer,3, [{file, "src/dcp_replicator.erl"}, {line,76}]}, {dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file, "src/dcp_replicator.erl"}, {line,336}]}]}}]}, [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"}, {line,1470}]}, {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,357}]}, {dcp_replicator,handle_info,2, [{file,"src/dcp_replicator.erl"}, {line,137}]}, {gen_server,try_dispatch,4, [{file,"gen_server.erl"}, {line,1123}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,1200}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,240}]}]}, {gen_server,call, [<34927.31455.58>,get_partitions, infinity]}}, {gen_server,call, ['dcp_replication_manager-bucket-12', {get_replicator_pid,250}, infinity]}}, {gen_server,call, [{'janitor_agent-bucket-12', 'ns_1@ec2-34-203-199-75.compute-1.amazonaws.com'}, {if_rebalance,<0.10696.204>, {wait_seqno_persisted,979,1203}}, infinity]}}}}]}, [{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,474}]}, {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,225}]}]}}}}.Rebalance Operation Id = d3d4899324f090bcea7f27d27ff71fca |
TAF Script to reproduce
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/al2023-p0-os_certify-vset00-00-n1ql_txns/testexec.14850.ini -p rerun=False,data_spec_name=initial_load_N1ql,txntimeout=5,index_quota_percent=40,kv_quota_percent=50,get-cbcollect-info=True,get-cbcollect-info=True,get-cbcollect-info=True,hostname=true,upgrade_version=7.2.5-7595,sirius_url=http://172.23.120.103:4000 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=single_bucket.def_scope_fifty_collections,N1ql_txn=True,data_load_spec=initial_load_N1ql,quota_percent=80,GROUP=rebalance_with_collection_crud'
Job name : al2023-os-certify-n1ql-txns