Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-5017
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 3
Iteration : 1st
In the 1st iteration of the longevity system test, seeing 8 rebalance failures due to the below mentioned errors. The error is similar to what was seen in MB-44949. But since there are 2 indexes (bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ) repeated in most of these error messages, it would be good to see if the underlying cause of these issues is same as MB-44949 or is this a different issue.
user:error,2021-04-24T10:47:42.187-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket5:idx3_GKkIg bucket6:idx2_T41NLfu bucket4:idx1_z8ygU1K bucket6:idx3_pMkC6N bucket5:idx1_bgMbnx].">>}}}}}.
|
--
|
[user:error,2021-04-24T11:12:53.687-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_tJe8P bucket4:idx1_sVdSC bucket4:idx2_MHP3y bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
--
|
[user:error,2021-04-24T11:34:49.770-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
--
|
[user:error,2021-04-24T11:58:21.839-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
--
|
[user:error,2021-04-24T12:27:24.568-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
--
|
[user:error,2021-04-24T12:53:35.558-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
--
|
[user:error,2021-04-24T13:23:02.836-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
--
|
[user:error,2021-04-24T13:48:14.539-07:00,ns_1@172.23.108.103:<0.23040.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,index,
|
<<"indexer rebalance failure - index build is in progress for indexes: [bucket7:idx2_eWDiG bucket7:idx1_Nz4KQ].">>}}}}}.
|
Indexer nodes in the cluster around the time of the first 2-3 rebalance failures : 172.23.104.67 172.23.105.107 172.23.121.117 172.23.96.252 172.23.96.253 172.23.99.11
Attached logs are collected between 1-2 PM on 4/24/21. If you need logs from earlier or later timestamps, pls let me know