Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35138

[System Test] : Seeing java.lang.IllegalStateException... read buffers exceeded on an analytics node after service start

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-3748
      Test : -test tests/analytics/test_analytics_rebalance.yml -scope tests/analytics/scope_analytics_rebalance.yml
      Scale : 4
      Iteration : 1st

      On 172.23.96.214, once service is restarted after the crashes reported in MB-35137, seeing the following on 172.23.96.214

      2019-07-16T16:20:55.225-07:00 INFO CBAS.bootstrap.AnalyticsNCApplication [main] Starting Couchbase Analytics driver, version: {"build.date":"Fri Jul 12 03:09:44 UTC 2019","build.number":"3748","build.version":"6.5.0-3748"}
      		 
      2019-07-16T16:21:49.580-07:00 ERRO CBAS.tcp.TCPEndpoint [TCPEndpoint IO Thread [/0.0.0.0:9116]] Unexpected tcp io error in connection TCPConnection[Remote Address: /172.23.96.216:9116 Local Address: /0.0.0.0:9116]
      		 
      java.lang.IllegalStateException: Channel:0[localClose: true localCloseAck: true remoteClose: false remoteCloseAck:false readCredits: 0 writeCredits: 0] read buffers exceeded
      		 
      	at org.apache.hyracks.net.protocols.muxdemux.FullFrameChannelReadInterface.read(FullFrameChannelReadInterface.java:72) ~[hyracks-net.jar:6.5.0-3748]
      		 
      	at org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.read(ChannelControlBlock.java:95) ~[hyracks-net.jar:6.5.0-3748]
      		 
      	at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.driveReaderStateMachine(MultiplexedConnection.java:417) ~[hyracks-net.jar:6.5.0-3748]
      		 
      	at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.notifyIOReady(MultiplexedConnection.java:119) ~[hyracks-net.jar:6.5.0-3748]
      		 
      	at org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:197) [hyracks-net.jar:6.5.0-3748]

      This is followed by a series of messages - Restarting process to ensure data integrity - in the analytics error logs, and finally another crash.

      Attachments

        Issue Links

          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-35565 Backport - [System Test] : Seeing java.lang.IllegalStateException... read buffers exceeded on an analytics node after service start

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-35137 [System Test] Analytics service crashed on 3 nodes after a successful swap rebalance

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link additional networking logs commit

          Web Link Fix commit

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty