Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.3
-
Untriaged
-
-
Unknown
-
CX Sprint 168, CX Sprint 169
Description
Build : 6.0.3-2880
Test : -test tests/integration/test_allFeatures_alice_timers.yml -scope tests/integration/scope_Xattrs_Alice.yml
Scale : 3
Day : 1st
Seeing the following on 172.23.96.214
2019-09-19T08:35:07.553-07:00 ERRO CBAS.storage.IndexCheckpoint [Executor-2208:79f625d9a7f0c3b72a16c0332c69f9d2] low watermark 0 less than the latest checkpoint low watermark {"id":107,"validComponentSequence":106,"lowWatermark":268503558,"lastComponentId":106,"masterNodeFlushMap":{"-1":-1}}
|
2019-09-19T08:35:07.565-07:00 ERRO CBAS.impls.LSMHarness [Executor-2208:79f625d9a7f0c3b72a16c0332c69f9d2] FLUSH operation.afterFinalize failed on {"class" : "LSMBTree", "dir" : "/data/@analytics/v_iodevice_4/storage/partition_12/Default/ds1/0/idx_result1", "memory" : [{"class":"LSMBTreeMemoryComponent", "state":"READABLE_UNWRITABLE_FLUSHING", "writers":0, "readers":1, "pendingFlushes":0, "id":"[107,107]"}, {"class":"LSMBTreeMemoryComponent", "state":"READABLE_WRITABLE", "writers":0, "readers":0, "pendingFlushes":0, "id":"[108,108]"}], "disk" : 2, "num-scheduled-flushes":1, "current-memory-component":1}
|
java.lang.IllegalStateException: Low watermark should always be increasing
|
at org.apache.asterix.common.storage.IndexCheckpoint.next(IndexCheckpoint.java:65) ~[asterix-common.jar:6.0.3-2880]
|
at org.apache.asterix.app.nc.IndexCheckpointManager.flushed(IndexCheckpointManager.java:87) ~[asterix-app.jar:6.0.3-2880]
|
at org.apache.asterix.common.ioopcallbacks.LSMIOOperationCallback.addComponentToCheckpoint(LSMIOOperationCallback.java:136) ~[asterix-common.jar:6.0.3-2880]
|
at org.apache.asterix.common.ioopcallbacks.LSMIOOperationCallback.afterFinalize(LSMIOOperationCallback.java:123) ~[asterix-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.doIo(LSMHarness.java:544) [hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.flush(LSMHarness.java:513) [hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.flush(LSMTreeIndexAccessor.java:122) [hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.FlushOperation.call(FlushOperation.java:38) [hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.FlushOperation.call(FlushOperation.java:29) [hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
|
at java.lang.Thread.run(Thread.java:834) [?:?]
|
2019-09-19T08:35:07.699-07:00 ERRO CBAS.nc.HaltCallback [Executor-2208:79f625d9a7f0c3b72a16c0332c69f9d2] Operation org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeFlushOperation@ddadf2d4 has failed
|
java.lang.IllegalStateException: Low watermark should always be increasing
|
at org.apache.asterix.common.storage.IndexCheckpoint.next(IndexCheckpoint.java:65) ~[asterix-common.jar:6.0.3-2880]
|
at org.apache.asterix.app.nc.IndexCheckpointManager.flushed(IndexCheckpointManager.java:87) ~[asterix-app.jar:6.0.3-2880]
|
at org.apache.asterix.common.ioopcallbacks.LSMIOOperationCallback.addComponentToCheckpoint(LSMIOOperationCallback.java:136) ~[asterix-common.jar:6.0.3-2880]
|
at org.apache.asterix.common.ioopcallbacks.LSMIOOperationCallback.afterFinalize(LSMIOOperationCallback.java:123) ~[asterix-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.doIo(LSMHarness.java:544) ~[hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.flush(LSMHarness.java:513) ~[hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.flush(LSMTreeIndexAccessor.java:122) ~[hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.FlushOperation.call(FlushOperation.java:38) ~[hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at org.apache.hyracks.storage.am.lsm.common.impls.FlushOperation.call(FlushOperation.java:29) ~[hyracks-storage-am-lsm-common.jar:6.0.3-2880]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
|
at java.lang.Thread.run(Thread.java:834) [?:?]
|
2019-09-19T08:35:08.666-07:00 FATA CBAS.util.ExitUtil [Executor-2208:79f625d9a7f0c3b72a16c0332c69f9d2] JVM halting with status 88; thread dump at halt:
|
"main" [tid=1 state=WAITING lock=java.util.concurrent.Semaphore$NonfairSync@31f4f75b]
|
at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
|
at java.base@11.0.4/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
|
at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
|
at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
|
at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
|
at java.base@11.0.4/java.util.concurrent.Semaphore.acquire(Semaphore.java:318)
|
at app//com.couchbase.analytics.control.AnalyticsDriver.main(AnalyticsDriver.java:109)
|
The test was rebalancing out an Eventing node at that time.
Attachments
Issue Links
- is triggered by
-
-
- Closed
-
- links to