Details
-
Bug
-
Resolution: Fixed
-
Critical
-
CBAS DP3
-
Untriaged
-
-
No
-
CX Sprint 65, CX Sprint 67
Description
Analytics Build : 5.0.0-639
We have a new longevity test that does the following:
1. 4 node cluster = 2 kv, 2 analytics
2. Create 4 CB buckets
3. Create 4 corresponding CBAS buckets
4. Load some docs in all buckets for sometime
5. Create 4 corresponding CBAS buckets, datasets and connect to the CB buckets
6. Run KV ops (with creates, deletes, updates) continuously on all buckets
7. Run analytic queries that touches both the analytics nodes and covers all datasets continuously. I will deliberately pump in queries with a small batch size so as to not run into query rejections due to full queue.
8. Stop after a specified time.
This test was run for 10 hrs. The analytics logs on both the nodes have tons of these kind of errors :
2017-08-09T18:30:43.289-07:00 ERRO CBAS.server.HttpServerHandler [nioEventLoopGroup-3-1] Failure handling HTTP Request |
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1862270983, max: 1864368128) |
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:624) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:578) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:709) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:698) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.PoolArena.allocate(PoolArena.java:213) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:262) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:131) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.1.6.Final.jar:4.1.6.Final] |
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] |
Looks like analytics need to keep aside some memory for query execution so in order to not run into these errors.
Attached analytics logs from both the nodes. cbcollectinfo location mentioned below.
The env is live if you want to take a look :
http://172.23.108.162:8091/ - KV node1
http://172.23.108.17:8091/ - CBAS node1
http://172.23.108.46:8091/ - CBAS node2