Details
Major Description
Build : 6.5.0-4947 (RC3)
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1st (day 1)
In the system longevity test, there is a step to failover a KV node and rebalance it out.
[2019-12-12T04:54:43-08:00, sequoiatools/couchbase-cli:6.5:250d02] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --force
|
[2019-12-12T04:55:15-08:00, sequoiatools/couchbase-cli:6.5:384eef] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
Around this time, on an analytics node 172.23.106.188, there are some errors seen in the logs:
2019-12-12T04:55:02.581-08:00 ERRO CBAS.conductor.Fixer [FixerThread:cbas:DISTRICT:c410e9f19382f3730899b16d231f1dd0:0] FixerThread:cbas:DISTRICT:c410e9f19382f3730899b16d231f1dd0:0 failed to refresh configurations
|
java.util.concurrent.CancellationException: null
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.cancel(...)(Unknown Source) ~[core-io-1.7.10.jar:?]
|
2019-12-12T04:55:02.581-08:00 ERRO CBAS.conductor.Fixer [FixerThread:cbas:DISTRICT:c410e9f19382f3730899b16d231f1dd0:2] FixerThread:cbas:DISTRICT:c410e9f19382f3730899b16d231f1dd0:2 failed to refresh configurations
|
java.util.concurrent.CancellationException: null
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.cancel(...)(Unknown Source) ~[core-io-1.7.10.jar:?]
|
2019-12-12T04:55:02.585-08:00 ERRO CBAS.conductor.Fixer [FixerThread:cbas:DISTRICT:c410e9f19382f3730899b16d231f1dd0:1] FixerThread:cbas:DISTRICT:c410e9f19382f3730899b16d231f1dd0:1 failed to refresh configurations
|
java.util.concurrent.CancellationException: null
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.cancel(...)(Unknown Source) ~[core-io-1.7.10.jar:?]
|
2019-12-12T04:55:33.716-08:00 ERRO CBAS.tcp.TCPEndpoint [TCPEndpoint IO Thread [/0.0.0.0:9116]] Unexpected tcp io error in connection TCPConnection[Remote Address: /172.23.104.61:9116 Local Address: /0.0.0.0:9116]
|
java.lang.IllegalStateException: Channel:196[localClose: true localCloseAck: true remoteClose: false remoteCloseAck:false readCredits: 0 writeCredits: 0] read buffers exceeded
|
at org.apache.hyracks.net.protocols.muxdemux.FullFrameChannelReadInterface.read(FullFrameChannelReadInterface.java:80) ~[hyracks-net.jar:6.5.0-4947]
|
at org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.read(ChannelControlBlock.java:96) ~[hyracks-net.jar:6.5.0-4947]
|
at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.driveReaderStateMachine(MultiplexedConnection.java:418) ~[hyracks-net.jar:6.5.0-4947]
|
at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.notifyIOReady(MultiplexedConnection.java:119) ~[hyracks-net.jar:6.5.0-4947]
|
at org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:199) [hyracks-net.jar:6.5.0-4947]
|
A similar issue was filed earlier MB-35138, and it was resolved by adding more logging to be able to debug this issue further. However the issue wasn't seen in the system test runs until now.
Attachments
Issue Links
- is duplicated by
-
MB-37393 [System test]: Analytics rebalance failed with service_error
-
- Closed
-
- links to