Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32775

[CX] Need back-off for node registration error messages in the log

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 136, CX Sprint 137

    Description

      The IllegalStateException repeats a lot in the log.

      2019-01-14T07:05:34.066Z ERRO CBAS.impl.IPCConnectionManager [IPC Network Listener Thread [/0.0.0.0:9115]] Exception processing message
      		 
      java.lang.IllegalStateException: Unknown pending node registration 2 for CC:0
      		 
              at org.apache.hyracks.control.nc.NodeControllerService.getPendingNodeRegistration(NodeControllerService.java:454) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.control.nc.NodeControllerService.setNodeRegistrationResult(NodeControllerService.java:438) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.control.nc.NodeControllerIPCI.deliverIncomingMessage(NodeControllerIPCI.java:92) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCSystem.deliverIncomingMessage(IPCSystem.java:111) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCHandle.processIncomingMessages(IPCHandle.java:171) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.read(IPCConnectionManager.java:376) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.processSelectedKeys(IPCConnectionManager.java:213) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:199) [hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181) [hyracks-ipc.jar:6.0.0-1693]
      		 
      2019-01-14T07:05:34.066Z ERRO CBAS.impl.IPCConnectionManager [IPC Network Listener Thread [/0.0.0.0:9115]] Exception processing message
      		 
      java.lang.IllegalStateException: Unknown pending node registration 2 for CC:0
      		 
              at org.apache.hyracks.control.nc.NodeControllerService.getPendingNodeRegistration(NodeControllerService.java:454) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.control.nc.NodeControllerService.setNodeRegistrationResult(NodeControllerService.java:438) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.control.nc.NodeControllerIPCI.deliverIncomingMessage(NodeControllerIPCI.java:92) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCSystem.deliverIncomingMessage(IPCSystem.java:111) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCHandle.processIncomingMessages(IPCHandle.java:171) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.read(IPCConnectionManager.java:376) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.processSelectedKeys(IPCConnectionManager.java:213) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:199) [hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181) [hyracks-ipc.jar:6.0.0-1693]
      		 
      2019-01-14T07:05:34.066Z ERRO CBAS.impl.IPCConnectionManager [IPC Network Listener Thread [/0.0.0.0:9115]] Exception processing message
      		 
      java.lang.IllegalStateException: Unknown pending node registration 2 for CC:0
      		 
              at org.apache.hyracks.control.nc.NodeControllerService.getPendingNodeRegistration(NodeControllerService.java:454) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.control.nc.NodeControllerService.setNodeRegistrationResult(NodeControllerService.java:438) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.control.nc.NodeControllerIPCI.deliverIncomingMessage(NodeControllerIPCI.java:92) ~[hyracks-control-nc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCSystem.deliverIncomingMessage(IPCSystem.java:111) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCHandle.processIncomingMessages(IPCHandle.java:171) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.read(IPCConnectionManager.java:376) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.processSelectedKeys(IPCConnectionManager.java:213) ~[hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:199) [hyracks-ipc.jar:6.0.0-1693]
      		 
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181) [hyracks-ipc.jar:6.0.0-1693]
      		 
      2019-01-14T07:05:34.067Z ERRO CBAS.impl.IPCConnectionManager [IPC Network Listener Thread [/0.0.0.0:9115]] Exception processing message

      Multiple log message in a millisecond.
      We need to implement some back-off mechanism for logging these errors.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-33260 [CX] Busy-loop reprocessing message in IPC

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link Change Review

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              dmitry.lychagin Dmitry Lychagin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty