Details
Critical Description
Observed in a recent upgrade test run[1], upon shutdown after the executor has been shutdown, the NC gets into busy loop seemingly attempting to process a heartbeat ack over and over again:
2019-03-06T05:30:53.631Z ERRO CBAS.impl.IPCConnectionManager [IPC Network Listener Thread [/0.0.0.0:9115]] Exception processing message
|
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4d3a8095 rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@6b1a85db[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 68]
|
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_191]
|
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_191]
|
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_191]
|
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_191]
|
at org.apache.hyracks.control.nc.NodeControllerIPCI.deliverIncomingMessage(NodeControllerIPCI.java:145) ~[hyracks-control-nc.jar:6.0.1-2037]
|
at org.apache.hyracks.ipc.impl.IPCSystem.deliverIncomingMessage(IPCSystem.java:111) ~[hyracks-ipc.jar:6.0.1-2037]
|
at org.apache.hyracks.ipc.impl.IPCHandle.processIncomingMessages(IPCHandle.java:171) ~[hyracks-ipc.jar:6.0.1-2037]
|
at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.read(IPCConnectionManager.java:376) ~[hyracks-ipc.jar:6.0.1-2037]
|
at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.processSelectedKeys(IPCConnectionManager.java:213) ~[hyracks-ipc.jar:6.0.1-2037]
|
at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:199) [hyracks-ipc.jar:6.0.1-2037]
|
at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181) [hyracks-ipc.jar:6.0.1-2037]
|
The CC (0014*) has restarted at the same time as this NC (0015*) attempts to shutdown, so these cannot be new messages that are being processed.
We should determine why this message is being reprocessed, and why IPC is preventing server shutdown.
[1] - http://analytics.jenkins.couchbase.com/job/cbas-cbcluster-docker-test/2653/
Attachments
Issue Links
- duplicates
-
MB-32775 [CX] Need back-off for node registration error messages in the log
-
- Closed
-