Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
This issue is branched from MB-45869, please take logs from that ticket.
Rebalance at 2021-04-21T04:04:51.677-07:00 fails because of PrepareTopologyChange timeout:
2861252 [user:error,2021-04-21T08:41:40.426-07:00,ns_1@172.23.108.103:<0.23324.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,eventing,
|
2861253 {agent_died,<30247.24974.0>,
|
2861254 {linked_process_died,<30247.30085.1993>,
|
2861255 {timeout,
|
2861256 {gen_server,call,
|
2861257 [<30247.25022.0>,
|
2861258 {call,"ServiceAPI.PrepareTopologyChange",
|
2861259 #Fun<json_rpc_connection.0.77329884>},
|
2861260 60000]}}}}}.
|
Which is a call to eventing on 172.23.123.27
On 172.23.123.27 I see that eventing created a lot (~2000) of streaming connections against ns_server around that time, which I believe could lead to a timeout.
Please investigate why there are so many streaming connections.
Attachments
Issue Links
- relates to
-
MB-45869 [System Test][Analytics] Rebalance failed with error - java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active (missing nodes: [xxxx], state: ACTIVE)
- Closed
-
MB-46782 [BP 6.6.3][System Test][Analytics] Rebalance failed with error - java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active (missing nodes: [xxxx], state: ACTIVE)
- Closed
For Gerrit Dashboard: MB-45954 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
152530,1 | MB-45954: Close gocb cluster instance on error | unstable | eventing | Status: ABANDONED | 0 | 0 |
152544,9 | MB-45954: Cache gocb cluster handles | unstable | eventing | Status: MERGED | +2 | +1 |
152668,5 | MB-45954: Close connection when no function listening on source or metatdata bucket | unstable | eventing | Status: MERGED | +2 | +1 |
153182,3 | MB-45954: Fix race between closing channel and reading from socket | unstable | eventing | Status: MERGED | +2 | +1 |