Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34084

Plasma shouldn't report fatal error while warming up due to low memory

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Plasma reports a fatal error while recovering due to low memory conditions

      read failure in recovery: Timeout: Unable to evict enough memory 57858192
      

      And this results in Indexer cleaning it up. Plasma shouldn't treat this as fatal error.

      2019-05-03T05:46:12.189+00:00 [Error] plasmaSlice:NewplasmaSlice Id 0x63fcb0 IndexInstId 15685412117998097443 fatal error occured: Unable to initialize /opt/couchbase/var/lib/couchbase/data/@2i/<redacted>/mainIndex, err = <redacted>/Mainstore#15685412117998097443:0 : fatal: read failure in recovery: Timeout: Unable to evict enough memory 57858192
       
      2019-05-03T05:46:12.189+00:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 15685412117998097443 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
      2019-05-03T05:46:12.189+00:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
              InstId: 15685412117998097443
              Defn: DefnId: 12214602226524518489 Name: <redacted> Using: plasma Bucket: <redacted> IsPrimary: false NumReplica: 1 InstVersion: 0
                      SecExprs: <ud><redacted></ud>
                      Desc: [false false]
                      PartitionScheme: SINGLE
                      HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud><redacted></ud> RetainDeletedXATTR: false
              State: INDEX_STATE_ACTIVE
              RState: RebalActive
              Stream: MAINT_STREAM
              Version: 0
              ReplicaId: 0
              PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} partnDefn {0 0 [:9105]}
      2019-05-03T05:46:12.229+00:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9105]
      2019-05-03T05:46:12.229+00:00 [Info] Indexer::forceCleanupIndexPartition 15685412117998097443 0 mark metadata as deleted
      2019-05-03T05:46:12.229+00:00 [Info] ClustMgr:handleCleanupPartition&{{12214602226524518489 <redacted> 8e8bc2e8c4bad2fb4e4638e3f0b85061 false <redacted> N1QL SINGLE  <redacted> [false false] true false [] true 1 [] false 0 0 0 0 0 0 0 0 0 [] [] 0 0} 15685412117998097443 0 0 true}
      2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443 real instance id 0 partitions [0]
      2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 12214602226524518489 instance 15685412117998097443 partitions [0]
      2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443
      2019-05-03T05:46:12.230+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance.  Delete index 12214602226524518489
      2019-05-03T05:46:12.314+00:00 [Info] Indexer::forceCleanupIndexPartition Cleaning up data files for 15685412117998097443 0
      2019-05-03T05:46:12.374+00:00 [Info] Indexer::forceCleanupIndexPartition 15685412117998097443 0 Cleanup partition in-memory data structure
      2019-05-03T05:46:12.375+00:00 [Info] Indexer::forceCleanupIndexPartition 15685412117998097443 0 actually delete metadata
      2019-05-03T05:46:12.375+00:00 [Info] ClustMgr:handleCleanupPartition&{{12214602226524518489 cem_jl_links_fname_cov_idx plasma migcem_jl 8e8bc2e8c4bad2fb4e4638e3f0b85061 false <redacted> N1QL SINGLE  <redacted> [false false] true false [] true 1 [] false 0 0 0 0 0 0 0 0 0 [] [] 0 0} 15685412117998097443 0 0 false}
      2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443 real instance id 0 partitions [0]
      2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 12214602226524518489 instance 15685412117998097443 partitions [0]
      2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 12214602226524518489 instance id 15685412117998097443
      2019-05-03T05:46:12.375+00:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance.  Delete index 12214602226524518489
      2019-05-03T05:46:12.452+00:00 [Info] Indexer::initFromPersistedState Done cleanup for PartitionId: 0 Endpoints: [:9105]
      2019-05-03T05:46:12.482+00:00 [Info] Indexer::initFromPersistedState Skipping index instance 15685412117998097443
      

      Attachments

        For Gerrit Dashboard: MB-34084
        # Subject Branch Project Status CR V

        Activity

          People

            akhil.mundroy Akhil Mundroy
            deepkaran.salooja Deepkaran Salooja
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty