magma_gtest: MagmaHistoryTest.read_only fails intermittently

[ RUN      ] MagmaHistoryTest.readonly

[02:46:23.820][INFO]: [magma]Magma::SetHistoryRetentionSize changing history size window from 0 bytes to 18446744073709551615 bytes

[02:46:23.822][INFO]: [magma]Recovering with Config:{"ActivePurgersLimit":1,"BloomFilterAccuracy":0.99,"BloomFilterAccuracyForBottomLevel":0.95,"CheckpointCreationInterval":120000,"CheckpointCreationThreshold":0.2,"CompressionAlgo":"LZ4","DataCompressionAlgo":"Zstd","DiskSnapshotLockWaitTime":30,"DumpDebugStats":false,"EnableAutoCheckpointing":true,"EnableBlockCache":true,"EnableDirectIO":false,"EnableDirectIOWrite":true,"EnableFastOpen":true,"EnableFileHistograms":true,"EnableGroupCommit":false,"EnableHistoryCompaction":false,"EnableKeyPrefixMode":false,"EnableLSMTreeVerification":true,"EnableMetaForDocumentStore":true,"EnableSSTablePreallocation":true,"EnableUpdateStatusForSet":true,"EnableWAL":true,"EnableWALPreallocation":true,"ExpiryFragThreshold":0.25,"GroupCommitMaxSyncWaitDuration":20,"GroupCommitMaxTransactionCount":20,"GroupName":"default","HeartbeatInterval":0,"IOQueueDepth":1,"KVStorePurgerInterval":60,"KeyPrefixSize":32,"KeyTreeAlignBlocks":false,"KeyTreeBlockSize":4096,"KeyTreeIndexBlockSize":4096,"KeyTreeMaxIndexDepth":3,"LSDFragmentationRatio":0.25,"LSDNumLevels":5,"LSMBlockWritesThresholdRatio":8.0,"LSMLSDMaxSSTableSize":10,"LSMLevelSizeMultiplier":10,"LSMMaxBaseLevelSize":4194304,"LSMMaxLevel0CompactionTTL":300,"LSMMaxNumLevel0Tables":16,"LSMMaxSSTableSize":2097152,"LSMMinCompactSize":4931584,"LSMNumLevels":5,"LSMNumMemtables":2,"LSMSizeMultiplierBaseLevel":1,"LocalNumLevels":2,"LogLevel":"debug","MaxCheckpoints":10,"MaxCoroutineStackSize":262144,"MaxDeltaSSTableDynamicSizeRatio":0.025,"MaxKVStoreLSDBufferSize":8192,"MaxKVStores":1024,"MaxRecoveryBytes":67108864,"MaxSSTableDynamicSizeRatio":1e-07,"MaxWALSize":4294967296,"MaxWriteCacheSize":67108864,"MemoryQuota":671088640,"MinCheckpointCreationInterval":0,"MinCheckpointOverheadSizeThreshold":2097152,"NumCompactors":1,"NumFileHistogramShards":16,"NumFlushers":1,"Path":"readonly","ReadAheadSize":0,"ReadOnly":false,"SeqTreeAlignBlocks":false,"SeqTreeBlockSize":4096,"SeqTreeIndexBlockSize":4096,"SeqTreeMaxIndexDepth":3,"UseCheckpointCompression":false,"UseCompression":true,"UseStableCheckpoint":false,"WALBufferSize":2097152,"WALNumIORetries":16,"WALSegmentSize":67108864,"WriteCacheRatio":0.2}

[02:46:23.823][INFO]: [magma/kvstore-1/rev-000000001]KVStore::Open started name:kvstore-1/rev-000000001 path:readonly/kvstore-1/rev-000000001 configBasePath:readonly/kvstore-1/rev-000000001

[02:46:23.827][INFO]: [magma/kvstore-1/rev-000000001]KVStore::Open completed name:kvstore-1/rev-000000001 highSeqno:0 replayOffset:1:0 historyEnabledSeqno:0

[02:46:23.829][INFO]: [magma/kvstore-1/rev-000000001]KVStore:magma/kvstore-1/rev-000000001 current history mode:0 set at seqno:0 updated to mode:1 set at seqno:1

[02:46:23.876][CRITICAL]: [magma/kvstore-1/rev-000000001/localIndex]Error occured during memtable flush (NotFound: path:readonly/kvstore-1/rev-000000001/localIndex/sstable.11.data  msg:Unable to open file  error:No such file or directory. Closest non-empty parent directory:. ).

[02:46:23.876][CRITICAL]: [magma/kvstore-1/rev-000000001/localIndex]Error occured during memtable flush (NotFound: path:readonly/kvstore-1/rev-000000001/localIndex/sstable.12.data  msg:Unable to open file  error:No such file or directory. Closest non-empty parent directory:. ).

/home/couchbase/jenkins/workspace/magma-linux-fusion/magma/tests/history_test.cc:43: Failure

Value of: util.FlushKVStore(h.GetKVStoreID())

  Actual: false

Expected: true

path:readonly/kvstore-1/rev-000000001/localIndex/sstable.12.data  msg:Unable to open file  error:No such file or directory. Closest non-empty parent directory:. 

/home/couchbase/jenkins/workspace/magma-linux-fusion/magma/tests/history_test.cc:34: Failure

Value of: h.VerifyHistoryIterator(db)

  Actual: false

Expected: true

VerifyHistorySeqIterator missing items Got:0 != Expected:799

/home/couchbase/jenkins/workspace/magma-linux-fusion/magma/tests/history_test.cc:34: Failure

Value of: h.VerifyHistoryIterator(db)

  Actual: false

Expected: true

VerifyHistorySeqIterator missing items Got:0 != Expected:799

[02:46:23.878][INFO]: [magma]Closing magma db (thread pool refcount 1)

[02:46:23.878][INFO]: [magma/kvstore-1/rev-000000001]KVStore::Shutdown

[02:46:23.879][INFO]: [magma/kvstore-1/rev-000000001]KVStore::~KVStore Destructor completed

[02:46:23.879][INFO]: [magma]Completed closing magma db

[02:46:23.880][INFO]: [magma]Magma::SetHistoryRetentionSize changing history size window from 0 bytes to 10 bytes

[02:46:23.880][INFO]: [magma]Recovering with Config:{"ActivePurgersLimit":1,"BloomFilterAccuracy":0.99,"BloomFilterAccuracyForBottomLevel":0.95,"CheckpointCreationInterval":120000,"CheckpointCreationThreshold":0.2,"CompressionAlgo":"LZ4","DataCompressionAlgo":"Zstd","DiskSnapshotLockWaitTime":30,"DumpDebugStats":false,"EnableAutoCheckpointing":true,"EnableBlockCache":true,"EnableDirectIO":false,"EnableDirectIOWrite":true,"EnableFastOpen":true,"EnableFileHistograms":true,"EnableGroupCommit":false,"EnableHistoryCompaction":false,"EnableKeyPrefixMode":false,"EnableLSMTreeVerification":true,"EnableMetaForDocumentStore":true,"EnableSSTablePreallocation":true,"EnableUpdateStatusForSet":true,"EnableWAL":true,"EnableWALPreallocation":true,"ExpiryFragThreshold":0.25,"GroupCommitMaxSyncWaitDuration":20,"GroupCommitMaxTransactionCount":20,"GroupName":"default","HeartbeatInterval":0,"IOQueueDepth":1,"KVStorePurgerInterval":60,"KeyPrefixSize":32,"KeyTreeAlignBlocks":false,"KeyTreeBlockSize":4096,"KeyTreeIndexBlockSize":4096,"KeyTreeMaxIndexDepth":3,"LSDFragmentationRatio":0.25,"LSDNumLevels":5,"LSMBlockWritesThresholdRatio":8.0,"LSMLSDMaxSSTableSize":10,"LSMLevelSizeMultiplier":10,"LSMMaxBaseLevelSize":4194304,"LSMMaxLevel0CompactionTTL":300,"LSMMaxNumLevel0Tables":16,"LSMMaxSSTableSize":2097152,"LSMMinCompactSize":4931584,"LSMNumLevels":5,"LSMNumMemtables":2,"LSMSizeMultiplierBaseLevel":1,"LocalNumLevels":2,"LogLevel":"debug","MaxCheckpoints":10,"MaxCoroutineStackSize":262144,"MaxDeltaSSTableDynamicSizeRatio":0.025,"MaxKVStoreLSDBufferSize":8192,"MaxKVStores":1024,"MaxRecoveryBytes":67108864,"MaxSSTableDynamicSizeRatio":1e-07,"MaxWALSize":4294967296,"MaxWriteCacheSize":67108864,"MemoryQuota":671088640,"MinCheckpointCreationInterval":120000,"MinCheckpointOverheadSizeThreshold":2097152,"NumCompactors":1,"NumFileHistogramShards":16,"NumFlushers":1,"Path":"readonly","ReadAheadSize":0,"ReadOnly":true,"SeqTreeAlignBlocks":false,"SeqTreeBlockSize":4096,"SeqTreeIndexBlockSize":4096,"SeqTreeMaxIndexDepth":3,"UseCheckpointCompression":false,"UseCompression":true,"UseStableCheckpoint":false,"WALBufferSize":2097152,"WALNumIORetries":16,"WALSegmentSize":67108864,"WriteCacheRatio":0.2}

/home/couchbase/jenkins/workspace/magma-linux-fusion/magma/tests/history_test.cc:264: Failure

Expected equality of these values:

  db.GetOldestHistorySeqno(1)

    Which is: 0

  1

[02:46:23.880][INFO]: [magma]Closing magma db (thread pool refcount 0)

[02:46:23.881][INFO]: [magma]Completed closing magma db

[02:46:23.881][WARNING]: readonly TearDown MemAllocated 3248

[  FAILED  ] MagmaHistoryTest.readonly (61 ms)

MagmaHistoryTest.readonly