Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Enterprise Edition 7.1.0 build 1361
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
-
KV 2021-Oct-21
Description
Test -
-test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
|
Day - 1
Cycle - 1
Scale - 3
Test step -
Rebalance in fts node.
[2021-09-27T22:25:38-07:00, sequoiatools/couchbase-cli:7.0:8fed98] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts |
[2021-09-27T22:25:56-07:00, sequoiatools/couchbase-cli:7.0:a70242] rebalance -c 172.23.108.103:8091 -u Administrator -p password |
On 172.23.104.155
diag.log
2021-09-27T22:56:25.628-07:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.104.155) - Worker <0.1517.275> (for action {move,{762, |
['ns_1@172.23.99.25',undefined], |
['ns_1@172.23.99.25', |
'ns_1@172.23.108.103'], |
[]}}) exited with reason {unexpected_exit,
|
{'EXIT', |
<0.32187.274>, |
{{dcp_wait_for_data_move_failed,
|
"WAREHOUSE", |
762, |
'ns_1@172.23.99.25', |
['ns_1@172.23.108.103'], |
{error,
|
no_stats_for_this_vbucket}},
|
[{ns_single_vbucket_mover,
|
'-wait_dcp_data_move/5-fun-0-', |
5, |
[{file,
|
"src/ns_single_vbucket_mover.erl"}, |
{line,
|
483}]}, |
{proc_lib,
|
init_p,3, |
[{file,
|
"proc_lib.erl"}, |
{line,
|
234}]}]}}} |
2021-09-27T22:56:25.646-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.104.155) - Rebalance exited with reason {mover_crashed, |
{unexpected_exit,
|
{'EXIT',<0.32187.274>, |
{{dcp_wait_for_data_move_failed,"WAREHOUSE", |
762,'ns_1@172.23.99.25', |
['ns_1@172.23.108.103'], |
{error,no_stats_for_this_vbucket}},
|
[{ns_single_vbucket_mover,
|
'-wait_dcp_data_move/5-fun-0-',5, |
[{file,"src/ns_single_vbucket_mover.erl"}, |
{line,483}]}, |
{proc_lib,init_p,3, |
[{file,"proc_lib.erl"},{line,234}]}]}}}}. |
Rebalance Operation Id = 69447666ac5b51c2038091606d87cd66
|
Vbucket in question is 762.
On 172.23.99.25
memcached.log
Following warnings observed for (vb:762) at a similar timeframe compared to rebalance exit.
2021-09-27T22:56:25.616020-07:00 WARNING 1520: (WAREHOUSE) DCP (Producer) eq_dcpq:replication:ns_1@172.23.99.25->ns_1@172.23.108.103:WAREHOUSE - (vb:762) Stream closing, sent until seqno 0 remaining items 0, reason: The stream closed early because the conn was disconnected. |
2021-09-27T22:56:25.616167-07:00 INFO [(WAREHOUSE) magma_3/kvstore-763/rev-000000001]0x7f28b57ab100 Ending seqno iterator (1=>82822) with seqno 81567 restarted 0 times |
2021-09-27T22:56:25.616260-07:00 WARNING (WAREHOUSE) DCPBackfillBySeqnoDisk::complete(): (vb:762) backfill create ended prematurely as the associated stream is deleted by the producer conn; cancelled |
