Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.6.0
-
6.6.0-7873-enterprise
couchbase-transactions-1.1.0-SNAPSHOT.jar
java-client-3.0.6-SNAPSHOT.jar
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Build: 6.6.0-7873
Scenario:
- 3 node cluster, 3 couchbase buckets
+----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+| Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |+----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+| 172.23.98.215 | kv | 36.0655737705 | 4339630080 | 3580170240 | 79736832 / 3758092288 | 10214 / 20462 | 6.6.0-7873-enterprise || 172.23.123.111 | kv | 18.6868686869 | 4201672704 | 3366232064 | 30670848 / 3758092288 | 10278 / 20473 | 6.6.0-7873-enterprise || 172.23.123.116 | kv | 48.7046632124 | 4201672704 | 3422486528 | 116359168 / 3758092288 | 10020 / 20118 | 6.6.0-7873-enterprise |+----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ -
2020-07-13 10:48:20,966 | test | INFO | MainThread | [table_view:display:72] Bucket statistics+----------+---------+----------+------------+-----+-------+------------+----------+-----------+| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |+----------+---------+----------+------------+-----+-------+------------+----------+-----------+| bucket-0 | membase | 2 | none | 0 | 9920 | 1541406720 | 90595104 | 470784378 || bucket-1 | membase | 2 | none | 0 | 10673 | 1541406720 | 95380544 | 443875257 || bucket-2 | membase | 2 | none | 0 | 9919 | 1541406720 | 90737664 | 472684883 |+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
- Load initial docs into all 3 buckets using transaction
- Swap rebalance 172.23.123.111 <—> 172.23.123.115
Observation:
Swap rebalance at step#3 fails with reason "mover_crashed wait_seqno_persisted_failed bucket-1",
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.13721.0>,
|
{{wait_seqno_persisted_failed,"bucket-1",341, 92,
|
[{'ns_1@172.23.123.115',
|
{'EXIT',
|
{{{{{badmatch, [{<25251.5355.0>,
|
{done,exit,
|
{socket_closed, {gen_server,call, [<25251.5288.0>,
|
{setup_streams,[339,340,341]}, infinity]}},
|
[{gen_server,call,3, [{file,"gen_server.erl"}, {line,214}]},
|
{dcp_replicator, '-spawn_and_wait/1-fun-0-',1, [{file, "src/dcp_replicator.erl"}, {line,243}]}]}}]},
|
[{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"}, {line,1374}]},
|
{dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,265}]},
|
{dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]},
|
{gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,636}]},
|
{gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,665}]},
|
{proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,247}]}]},
|
{gen_server,call, [<25251.5287.0>,get_partitions, infinity]}},
|
{gen_server,call,
|
['dcp_replication_manager-bucket-1',
|
{get_replicator_pid,340}, infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-bucket-1',
|
'ns_1@172.23.123.115'},
|
{if_rebalance,<0.13522.0>,
|
{wait_seqno_persisted,341,92}}, infinity]}}}}]},
|
[{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [{file,"src/ns_single_vbucket_mover.erl"}, {line,488}]},
|
{proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}.
|
Rebalance Operation Id = 2c18c85d730aa4c35cdf7f72b414ddd4
|
|
|
Worker <0.13559.0> (for action {move,{341,
|
['ns_1@172.23.123.111',
|
'ns_1@172.23.98.215',
|
'ns_1@172.23.123.116'],
|
['ns_1@172.23.123.115',
|
'ns_1@172.23.98.215',
|
'ns_1@172.23.123.116'],
|
[]}}) exited with reason {unexpected_exit,
|
{'EXIT',
|
<0.13721.0>,
|
{{wait_seqno_persisted_failed,"bucket-1",341,92,
|
[{'ns_1@172.23.123.115',
|
{'EXIT',
|
{{{{{badmatch,[{<25251.5355.0>,
|
{done,exit,
|
{socket_closed,
|
{gen_server,call,
|
[<25251.5288.0>,
|
{setup_streams,
|
[339,340,341]},
|
infinity]}},
|
[{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},
|
{dcp_replicator,'-spawn_and_wait/1-fun-0-',1,[{file,"src/dcp_replicator.erl"},{line,243}]}]}}]},
|
[{misc,sync_shutdown_many_i_am_trapping_exits,1,[{file,"src/misc.erl"},{line,1374}]},
|
{dcp_replicator,spawn_and_wait,1,[{file,"src/dcp_replicator.erl"},{line,265}]},
|
{dcp_replicator,handle_call,3,[{file,"src/dcp_replicator.erl"},{line,121}]},
|
{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},
|
{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},
|
{gen_server,call,[<25251.5287.0>,get_partitions,infinity]}},
|
{gen_server,call,['dcp_replication_manager-bucket-1',
|
{get_replicator_pid,340},infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-bucket-1','ns_1@172.23.123.115'},
|
{if_rebalance,<0.13522.0>,
|
{wait_seqno_persisted,341,92}},infinity]}}}}]},
|
[{ns_single_vbucket_mover,'-wait_seqno_persisted_many/5-fun-2-',5,
|
[{file,"src/ns_single_vbucket_mover.erl"},{line,488}]},
|
{proc_lib,init_p,3,[{file,"proc_lib.erl"},{line, 232}]}]}}}
|
cbcollect logs:
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.111.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.115.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.123.116.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance_failure/collectinfo-2020-07-13T174841-ns_1%40172.23.98.215.zip
Attachments
Issue Links
- duplicates
-
-
- Closed
-