Details
-
Bug
-
Resolution: Won't Fix
-
Critical
-
6.0.0
-
Untriaged
-
-
Unknown
-
CX Sprint 118
Description
Build : 6.0.0-1592
Test : -test tests/analytics/test_analytics_rebalance.yml -scope tests/analytics/scope_analytics_rebalance.yml
Scale : 4
Iteration : 3rd
While running the DDL for disconnect link, analytics service crashed on 172.23.96.215. Seeing the following errors on 172.23.96.215 at the time of crash.
2018-09-01T20:16:00.391-07:00 DEBU CBAS.impls.LSMHarness [Executor-294:b98ab0cf6c527e87ea751777c38001d1] Started a merge operation for index: {"class" : "LSMBTree", "dir" : "/data/idx/@analytics/v_iodevice_0/storage/partition_24/Default/ds1/1/ds1", "memory" : [{"class":"LSMBTreeMemoryComponent", "state":"UNREADABLE_UNWRITABLE", "writers":0, "readers":1, "pendingFlushes":0, "id":"[17,17]"}, {"class":"LSMBTreeMemoryComponent", "state":"READABLE_WRITABLE", "writers":1, "readers":1, "pendingFlushes":0, "id":"[18,18]"}], "disk" : 5, "num-scheduled-flushes":1, "current-memory-component":1}
|
2018-09-01T20:16:00.394-07:00 DEBU CBAS.impls.LSMHarness [Executor-256:b98ab0cf6c527e87ea751777c38001d1] Started a merge operation for index: {"class" : "LSMBTree", "dir" : "/data/idx/@analytics/v_iodevice_6/storage/partition_22/Default/ds1/1/ds1", "memory" : [{"class":"LSMBTreeMemoryComponent", "state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[17,17]"}, {"class":"LSMBTreeMemoryComponent", "state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[18,18]"}], "disk" : 5, "num-scheduled-flushes":0, "current-memory-component":1}
|
2018-09-01T20:16:00.531-07:00 DEBU CBAS.work.WorkQueue [Worker:b98ab0cf6c527e87ea751777c38001d1] Executing: NotifyTaskComplete: [b98ab0cf6c527e87ea751777c38001d1[JID:3.32840:TAID:TID:ANID:ODID:5:0:14:0]
|
2018-09-01T20:16:00.718-07:00 DEBU CBAS.work.WorkQueue [Worker:b98ab0cf6c527e87ea751777c38001d1] Executing: AbortTasks
|
2018-09-01T20:16:00.719-07:00 DEBU CBAS.work.WorkQueue [Worker:b98ab0cf6c527e87ea751777c38001d1] Executing: CleanupJoblet
|
2018-09-01T20:16:00.719-07:00 DEBU CBAS.work.CleanupJobletWork [Worker:b98ab0cf6c527e87ea751777c38001d1] cleaning up after job: JID:3.32840
|
2018-09-01T20:16:00.725-07:00 DEBU CBAS.buffercache.BufferCache [Executor-433:b98ab0cf6c527e87ea751777c38001d1:JID:3.32840:TAID:TID:ANID:ODID:5:0:18:0:SuperActivityOperatorNodePushable] Failure while trying to read a page from disk
|
org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException
|
at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:51) ~[hyracks-api.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IoRequest.queue(IoRequest.java:105) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IoRequest.read(IoRequest.java:76) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IOManager.asyncRead(IOManager.java:318) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IOManager.syncRead(IOManager.java:249) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:553) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.buffercache.BufferCache.tryRead(BufferCache.java:524) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:194) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.btree.impls.DiskBTreeRangeSearchCursor.acquirePage(DiskBTreeRangeSearchCursor.java:98) ~[hyracks-storage-am-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.fetchNextLeafPage(BTreeRangeSearchCursor.java:104) ~[hyracks-storage-am-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.btree.impls.DiskBTreeRangeSearchCursor.doHasNext(DiskBTreeRangeSearchCursor.java:45) ~[hyracks-storage-am-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.EnforcedIndexCursor.hasNext(EnforcedIndexCursor.java:69) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor.pushIntoQueueFromCursorAndReplaceThisElement(LSMIndexSearchCursor.java:188) ~[hyracks-storage-am-lsm-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.pushOutputElementIntoQueueIfNeeded(LSMBTreeRangeSearchCursor.java:204) ~[hyracks-storage-am-lsm-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.checkPriorityQueue(LSMBTreeRangeSearchCursor.java:186) ~[hyracks-storage-am-lsm-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor.doHasNext(LSMIndexSearchCursor.java:138) ~[hyracks-storage-am-lsm-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.EnforcedIndexCursor.hasNext(EnforcedIndexCursor.java:69) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.doHasNext(LSMBTreeSearchCursor.java:60) ~[hyracks-storage-am-lsm-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.EnforcedIndexCursor.hasNext(EnforcedIndexCursor.java:69) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:241) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:290) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.open(EmptyTupleSourceRuntimeFactory.java:55) ~[algebricks-runtime.jar:6.0.0-1592]
|
at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$SourcePushRuntime.initialize(AlgebricksMetaOperatorDescriptor.java:105) ~[algebricks-runtime.jar:6.0.0-1592]
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212) ~[hyracks-api.jar:6.0.0-1592]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
|
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
|
Caused by: java.lang.InterruptedException
|
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_181]
|
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_181]
|
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[?:1.8.0_181]
|
at org.apache.hyracks.control.nc.io.IoRequest.queue(IoRequest.java:103) ~[hyracks-control-nc.jar:6.0.0-1592]
|
... 27 more
|
2018-09-01T20:16:00.733-07:00 DEBU CBAS.buffercache.BufferCache [Executor-422:b98ab0cf6c527e87ea751777c38001d1:JID:3.32840:TAID:TID:ANID:ODID:5:0:16:0:SuperActivityOperatorNodePushable] Failure while trying to read a page from disk
|
org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException
|
at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:51) ~[hyracks-api.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IoRequest.queue(IoRequest.java:105) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IoRequest.read(IoRequest.java:76) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IOManager.asyncRead(IOManager.java:318) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.io.IOManager.syncRead(IOManager.java:249) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:553) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.buffercache.BufferCache.tryRead(BufferCache.java:524) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:194) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.btree.impls.DiskBTreeRangeSearchCursor.acquirePage(DiskBTreeRangeSearchCursor.java:98) ~[hyracks-storage-am-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.fetchNextLeafPage(BTreeRangeSearchCursor.java:104) ~[hyracks-storage-am-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.btree.impls.DiskBTreeRangeSearchCursor.doHasNext(DiskBTreeRangeSearchCursor.java:45) ~[hyracks-storage-am-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.EnforcedIndexCursor.hasNext(EnforcedIndexCursor.java:69) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor.pushIntoQueueFromCursorAndReplaceThisElement(LSMIndexSearchCursor.java:188) ~[hyracks-storage-am-lsm-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.pushOutputElementIntoQueueIfNeeded(LSMBTreeRangeSearchCursor.java:204) ~[hyracks-storage-am-lsm-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.checkPriorityQueue(LSMBTreeRangeSearchCursor.java:186) ~[hyracks-storage-am-lsm-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor.doHasNext(LSMIndexSearchCursor.java:138) ~[hyracks-storage-am-lsm-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.EnforcedIndexCursor.hasNext(EnforcedIndexCursor.java:69) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.doHasNext(LSMBTreeSearchCursor.java:60) ~[hyracks-storage-am-lsm-btree.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.common.EnforcedIndexCursor.hasNext(EnforcedIndexCursor.java:69) ~[hyracks-storage-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:241) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:290) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.open(EmptyTupleSourceRuntimeFactory.java:55) ~[algebricks-runtime.jar:6.0.0-1592]
|
at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$SourcePushRuntime.initialize(AlgebricksMetaOperatorDescriptor.java:105) ~[algebricks-runtime.jar:6.0.0-1592]
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212) ~[hyracks-api.jar:6.0.0-1592]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
|
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
|
Caused by: java.lang.InterruptedException
|
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_181]
|
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_181]
|
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[?:1.8.0_181]
|
at org.apache.hyracks.control.nc.io.IoRequest.queue(IoRequest.java:103) ~[hyracks-control-nc.jar:6.0.0-1592]
|
... 27 more
|
To the test, the error that surfaced out is this -
[2018-09-01T19:48:21-07:00, sequoiatools/cbq:5fe325] -e=http://172.23.96.212:8095 -u=Administrator -p=password -script=disconnect link Local; -t 2m
|
→
|
|
|
Error occurred on container - sequoiatools/cbq:[-e=http://172.23.96.212:8095 -u=Administrator -p=password -script=disconnect link Local; -t 2m]
|
|
|
docker logs 5fe325
|
docker start 5fe325
|
|
|
@Connected to : http://172.23.96.212:8095/. Type Ctrl-D to exit.
|
|
|
-[31m ERROR 108 : N1QL: Connection failure {
|
6 "requestID": "6e4bf334-c887-49c2-b536-90939ff6838e",
|
"errors": [{
|
"code": 23000,
|
8 "msg": "Analytics Service is temporarily unavailable"
|
}],
|
"status": "fatal",
|
"metrics": {
|
"elapsedTime": "58.221364ms",
|
" "executionTime": "48.961981ms",
|
"resultCount": 0,
|
"resultSize": 0,
|
"processedObjects": 0,
|
"errorCount": 1
|
}
|
}
|
[0m
|
|
|
[pull] sequoiatools/cmd
|
[2018-09-01T20:16:36-07:00, sequoiatools/cmd:4ba750] 30
|