Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.6.0
-
Enterprise Edition 7.1.0 build 2440
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
TEST
-test tests/eventing/neo/test_eventing_rebalance_rbac.yml -scope tests/eventing/neo/scope_eventing_rebalance.yml
|
Day - 1
Cycle - 4
Scale - 3
TEST STEP
Rebalance out eventing node.
[2022-03-07T16:00:24-08:00, sequoiatools/couchbase-cli:7.1:eb4e62] rebalance -c 172.23.104.16:8091 --server-remove 172.23.104.23 -u Administrator -p password |
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.104.16:8091 --server-remove 172.23.104.23 -u Administrator -p password] |
|
docker logs eb4e62
|
docker start eb4e62
|
|
*Unable to display progress bar on this os |
JERROR: Rebalance failed. See logs for detailed reason. You can try again. |
FAILURE
Rebalance exited due to timeout.
2022-03-07T16:23:23.141-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.104.16) - Rebalance exited with reason {service_rebalance_failed,eventing, |
{worker_died,
|
{'EXIT',<0.2016.378>, |
{rebalance_failed,
|
{service_error,
|
<<"eventing rebalance hasn't made progress for past 1200 secs">>}}}}}. |
Rebalance Operation Id = d0468c56a828245da829b89b1c4946eb
|
OBSERVATION
From ns_server.eventing.log on 172.23.104.21 isRebalanceOngoing is false for approx. 20 minutes prior to rebalance failure.
2022-03-07T16:02:23.010-08:00 [Info] Consumer::RebalanceTaskProgress [worker_n1ql2_0_0:/tmp/127.0.0.1:8091_0_2471638815.sock:113831] isBootstrapping: false isRebalanceOngoing: false vbsRemainingToCloseStream len: 0 dump: [] vbsRemainingToStreamReq len: 0 dump: [] |
2022-03-07T16:02:23.010-08:00 [Info] Consumer::RebalanceTaskProgress [worker_n1ql2_0_0:/tmp/127.0.0.1:8091_0_2471638815.sock:113831] uuid: 778736fd979cb089694b9266cc4028f0 eject node UUIDs: [5a957ac3d16b08bab4324fce5a848948] |
2022-03-07T16:02:23.010-08:00 [Info] ServiceMgr::getRebalanceProgress Function: n1ql2_0 rebalance progress from node with rest port: 8091 progress: &{0 0 0 0 <nil>} err: <nil> |
List of eventing nodes
- 172.23.104.21
- 172.23.104.23
- 172.23.96.31
Attachments
Issue Links
- is a backport of
-
MB-51336 [System Test] Eventing rebalance failed due to timeout
- Closed