Details
-
Bug
-
Resolution: Fixed
-
Major
-
5.5.4, 6.0.1, 6.5.0
-
Untriaged
-
Unknown
Description
Plasma reports a fatal error while recovering due to low memory conditions
read failure in recovery: Timeout: Unable to evict enough memory 57858192
|
And this results in Indexer cleaning it up. Plasma shouldn't treat this as fatal error.
2019-05-03T05:46:12.189+00:00 [Error] plasmaSlice:NewplasmaSlice Id 0x63fcb0 IndexInstId 15685412117998097443 fatal error occured: Unable to initialize /opt/couchbase/var/lib/couchbase/data/@2i/<redacted>/mainIndex, err = <redacted>/Mainstore#15685412117998097443:0 : fatal: read failure in recovery: Timeout: Unable to evict enough memory 57858192
|
|
2019-05-03T05:46:12.189+00:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 15685412117998097443 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
|
2019-05-03T05:46:12.189+00:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
|
InstId: 15685412117998097443
|
Defn: DefnId: 12214602226524518489 Name: <redacted> Using: plasma Bucket: <redacted> IsPrimary: false NumReplica: 1 InstVersion: 0
|
SecExprs: <ud><redacted></ud>
|
Desc: [false false]
|
PartitionScheme: SINGLE
|
HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud><redacted></ud> RetainDeletedXATTR: false
|
State: INDEX_STATE_ACTIVE
|
RState: RebalActive
|
Stream: MAINT_STREAM
|
Version: 0
|
ReplicaId: 0
|
PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} partnDefn {0 0 [:9105]}
|
2019-05-03T05:46:12.229+00:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9105]
|
2019-05-03T05:46:12.229+00:00 [Info] Indexer::forceCleanupIndexPartition 15685412117998097443 0 mark metadata as deleted
|
2019-05-03T05:46:12.229+00:00 [Info] ClustMgr:handleCleanupPartition&{{12214602226524518489 <redacted> 8e8bc2e8c4bad2fb4e4638e3f0b85061 false <redacted> N1QL SINGLE <redacted> [false false] true false [] true 1 [] false 0 0 0 0 0 0 0 0 0 [] [] 0 0} 15685412117998097443 0 0 true}
|
2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443 real instance id 0 partitions [0]
|
2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 12214602226524518489 instance 15685412117998097443 partitions [0]
|
2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443
|
2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance. Delete index 12214602226524518489
|
2019-05-03T05:46:12.314+00:00 [Info] Indexer::forceCleanupIndexPartition Cleaning up data files for 15685412117998097443 0
|
2019-05-03T05:46:12.374+00:00 [Info] Indexer::forceCleanupIndexPartition 15685412117998097443 0 Cleanup partition in-memory data structure
|
2019-05-03T05:46:12.375+00:00 [Info] Indexer::forceCleanupIndexPartition 15685412117998097443 0 actually delete metadata
|
2019-05-03T05:46:12.375+00:00 [Info] ClustMgr:handleCleanupPartition&{{12214602226524518489 cem_jl_links_fname_cov_idx plasma migcem_jl 8e8bc2e8c4bad2fb4e4638e3f0b85061 false <redacted> N1QL SINGLE <redacted> [false false] true false [] true 1 [] false 0 0 0 0 0 0 0 0 0 [] [] 0 0} 15685412117998097443 0 0 false}
|
2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443 real instance id 0 partitions [0]
|
2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 12214602226524518489 instance 15685412117998097443 partitions [0]
|
2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443
|
2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance. Delete index 12214602226524518489
|
2019-05-03T05:46:12.452+00:00 [Info] Indexer::initFromPersistedState Done cleanup for PartitionId: 0 Endpoints: [:9105]
|
2019-05-03T05:46:12.482+00:00 [Info] Indexer::initFromPersistedState Skipping index instance 15685412117998097443
|