Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.1.0
Affects Version/s: 7.1.0
Component/s: storage-engine
Labels:
- magma
Environment:
7.1.0-1316

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://cb-engineering.s3.amazonaws.com/disk_crash.zip
Epic Link:
Magma GA
Story Points:
1
Is this a Regression?:
Unknown

Description

STEPS TO RECREATE:
DISK FULL TEST

Create a 4 node cluster
Create 5 million items (doc size = 2048) and replicas =1
Fill entire disk , ( "fallocate -l <space left on disk> <file_name>")
After Disk is full, Start doc ops create:upsert docs until ep_data_write_failed > 0(ensured using cbstats)
During doc ops keep killing memcached on all nodes (kill -9 $(pgrep memcached)
Observed Memcached crashed in magma::ThreadPool::GetQueueSize

BackTrace:
Core found on Node 172.23.104.216

(gdb) bt full

#0  _M_ptr (this=0x20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/thread_pool.cc:183

No locals.

#1  get (this=0x20) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/unique_ptr.h:422

No locals.

#2  getQueue (workerType=magma::ThreadPool::Flusher, this=0x0)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/thread_pool.cc:183

        queue = 0x0

#3  magma::ThreadPool::GetQueueSize(magma::ThreadPool::WorkerType) ()

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/thread_pool.cc:176

No locals.

#4  0x00000000008f0072 in magma::Magma::Impl::getStats(magma::Magma::MagmaStats&) ()

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db_stats.cc:304

        mstats = {NGets = 0, NSets = 14047, NInserts = 0, NDeletes = 0, NCommitBatches = 709, NFlushes = 0,

          NumNormalFlushes = 0, NumSyncFlushes = 0, NumPersistentFlushes = 0, NBloomFilterHits = 23409,

          NBloomFilterMisses = 246, BloomFilterFPR = 0.0103994925, BytesOutgoing = 0, NReadBytes = 1038820,

          NReadBytesCompact = 0, NReadBytesGet = 1038820, NReadIOs = 455, NReadIOsGet = 455, BytesIncoming = 29371601,

          NWriteBytes = 0, NWriteBytesCompact = 0, NWriteIOs = 0, NSyncs = 0, NGetStats = 0, NGetStatsComputed = 0,

          NCompacts = 0, NTTLCompacts = 0, NFileCountCompacts = 0, NWriterCompacts = 0, NTablesCreated = 0,

          NTablesDeleted = 0, NTableFiles = 1899, MemoryQuota = 109445120, NActivePurgers = 0, NPurgerRuns = 0,

          BlockCacheQuota = 25804992, BlockCacheMemUsed = 44460, WriteCacheQuota = 41328203, WriteCacheMemUsed = 35946496,

          BufferMemUsed = 0, HistogramMemUsed = 100728, LSMTreeObjectMemUsed = 770640, TotalMemUsed = 39376864,

          ActiveIndexBlocksSize = 3681406, ActiveBloomFilterMemUsed = 380411, ActiveDiskUsage = 515215360,

          TotalIndexBlocksSize = 5498209, TotalBloomFilterMemUsed = 380411, BloomFilterMemUsedRatio = 0.00347581506,

          TotalDiskUsage = 768700416, TableObjectMemUsed = 518816, TableMetaMemUsed = 605809, WALMemUsed = 65616,

          WALBufferMemUsed = 0, TreeSnapshotMemoryUsed = 943888, ReadAheadBufferMemUsed = 0, ActiveDataSize = 515215360,

          LogicalDataSize = 517584773, LogicalDiskSize = 517584773, Fragmentation = 0,

          FragmentationWithOverhead = 0.329757929, BlockCacheHits = 37, BlockCacheMisses = 5149,

          BlockCacheHitRatio = 0.713459313, FlushQueueSize = 0, CompactQueueSize = 0, WriteAmp = 0, ReadAmp = 0,

          ReadAmpGet = 0, ReadIOAmp = 0, BytesPerRead = 0, NGetsPerSec = 0, NSetsPerSec = 0, NInsertsPerSec = 0,

          NBloomFilterHitsPerSec = 0, NBloomFilterMissesPerSec = 0, BlockCacheHitsPerSec = 0, BlockCacheMissesPerSec = 0,

          RecentBlockCacheHitRatio = 0, NDeletesPerSec = 0, NCommitBatchesPerSec = 0, NFlushesPerSec = 0,

          NCompactsPerSec = 0, NGetStatsPerSec = 0, NGetStatsComputedPerSec = 0, NSyncsPerSec = 0, NReadBytesPerSec = 0,

---Type <return> to continue, or q <return> to quit---

          NReadBytesGetPerSec = 0, NReadBytesCompactPerSec = 0, BytesOutgoingPerSec = 0, NReadIOsPerSec = 0,

          NReadIOsGetPerSec = 0, BytesIncomingPerSec = 0, NWriteBytesPerSec = 0, NWriteIOsPerSec = 0,

          NWriteBytesCompactPerSec = 0, RecentWriteAmp = 0, RecentReadAmp = 0, RecentReadAmpGet = 0, RecentReadIOAmp = 0,

          RecentBytesPerRead = 0, RecentBloomFilterFPR = 0, ReplayWALOffset = {static npos = 18446744073709551615,

            _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7feed7ee2768 "60:5971968"}, _M_string_length = 10, {_M_local_buf = "60:5971968\000\000\000\000\000",

              _M_allocated_capacity = 4121135852866383926}}, IndexResidentRatio = 0.00151862879, KeyStats = {UID = {

              static npos = 18446744073709551615,

              _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7feed7ee2790 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}},

            NItemsPut = 14047, NItemsDel = 0, NItemsGet = 14047, NBloomFilterHits = 23409, NBloomFilterMisses = 246,

            BytesIncoming = 688303, BytesOutgoing = 0, NWriteBytes = 0, NReadBytes = 1038820, NSyncs = 0, NReadIOs = 455,

            NWriteIOs = 0, NItemsGarbageCollected = 0, NItemsMerged = 0, NItemsTransformed = 0, NItemsDropped = 0,

            NTablesCreated = 0, NTablesDeleted = 0, NTableFiles = 835, NCompacts = 0, NTTLCompacts = 0,

            NFileCountCompacts = 0, NWriterCompacts = 0, ActiveDiskUsage = 22335488, ActiveBloomFilterMemUsed = 377906,

            ActiveIndexBlocksSize = 176825, TotalDiskUsage = 22335488, TotalBloomFilterMemUsed = 377906,

            TotalIndexBlocksSize = 176825, TableObjectMemUsed = 207080, TableMetaMemUsed = 253840,

            MemtablesMemUsed = 1589248, BufferMemUsed = 0, ReadAheadBufferMemUsed = 0, NFailedCompacts = 0,

            TreeSnapshotMemoryUsed = 366768, PendingCheckpointBytes = 0, CheckpointOverhead = 0, WriteAmp = 0, ReadAmp = 0,

            ReadIOAmp = 0.032391257919840537, BloomFilterFPR = 0.010399492707672797, LogicalDataSize = 23990096,

            LogicalDiskSize = 23990096, Writer = {NInserts = 0, NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0,

              NSyncs = 0, NDataBlockWrites = 0, NIndexBlockWrites = 0}, Reader = {NBytesOutgoing = 0, NReadBytes = 1038820,

              NReadIOs = 455, NDataBlockReads = 246, NIndexBlockReads = 246}, CompactWriter = {NInserts = 0,

              NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0, NSyncs = 0, NDataBlockWrites = 0, NIndexBlockWrites = 0},

            CompactReader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0},

            IterateReader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0},

            LevelStats = {<std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >> = {

                _M_impl = {<std::allocator<magma::LSMTreeLevelStats>> = {<__gnu_cxx::new_allocator<magma::LSMTreeLevelStats>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >::_Vector_impl_data> = {_M_start = 0x7feed407e380, _M_finish = 0x7feed407e510,

                    _M_end_of_storage = 0x7feed407e510}, <No data fields>}}, <No data fields>}}, SeqStats = {UID = {

              static npos = 18446744073709551615,

---Type <return> to continue, or q <return> to quit---

              _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7feed7ee2a10 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}},

            NItemsPut = 14047, NItemsDel = 0, NItemsGet = 0, NBloomFilterHits = 0, NBloomFilterMisses = 0,

            BytesIncoming = 29175619, BytesOutgoing = 0, NWriteBytes = 0, NReadBytes = 0, NSyncs = 0, NReadIOs = 0,

            NWriteIOs = 0, NItemsGarbageCollected = 0, NItemsMerged = 0, NItemsTransformed = 0, NItemsDropped = 0,

            NTablesCreated = 0, NTablesDeleted = 0, NTableFiles = 229, NCompacts = 0, NTTLCompacts = 0,

            NFileCountCompacts = 0, NWriterCompacts = 0, ActiveDiskUsage = 486039552, ActiveBloomFilterMemUsed = 0,

            ActiveIndexBlocksSize = 3469511, TotalDiskUsage = 739524608, TotalBloomFilterMemUsed = 0,

            TotalIndexBlocksSize = 5286314, TableObjectMemUsed = 104656, TableMetaMemUsed = 119004,

            MemtablesMemUsed = 33292288, BufferMemUsed = 0, ReadAheadBufferMemUsed = 0, NFailedCompacts = 0,

            TreeSnapshotMemoryUsed = 297712, PendingCheckpointBytes = 0, CheckpointOverhead = 0, WriteAmp = 0, ReadAmp = 0,

            ReadIOAmp = 0, BloomFilterFPR = 0, LogicalDataSize = 486058963, LogicalDiskSize = 486058963, Writer = {

              NInserts = 0, NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0, NSyncs = 0, NDataBlockWrites = 0,

              NIndexBlockWrites = 0}, Reader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0,

              NIndexBlockReads = 0}, CompactWriter = {NInserts = 0, NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0,

              NSyncs = 0, NDataBlockWrites = 0, NIndexBlockWrites = 0}, CompactReader = {NBytesOutgoing = 0, NReadBytes = 0,

              NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0}, IterateReader = {NBytesOutgoing = 0, NReadBytes = 0,

              NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0},

            LevelStats = {<std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >> = {

                _M_impl = {<std::allocator<magma::LSMTreeLevelStats>> = {<__gnu_cxx::new_allocator<magma::LSMTreeLevelStats>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >::_Vector_impl_data> = {_M_start = 0x7feed407e540, _M_finish = 0x7feed407e6d0,

                    _M_end_of_storage = 0x7feed407e6d0}, <No data fields>}}, <No data fields>}}, LocalStats = {UID = {

              static npos = 18446744073709551615,

              _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7feed7ee2c90 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}},

            NItemsPut = 1418, NItemsDel = 0, NItemsGet = 498, NBloomFilterHits = 0, NBloomFilterMisses = 0,

            BytesIncoming = 364546, BytesOutgoing = 0, NWriteBytes = 0, NReadBytes = 0, NSyncs = 0, NReadIOs = 0,

            NWriteIOs = 0, NItemsGarbageCollected = 0, NItemsMerged = 0, NItemsTransformed = 0, NItemsDropped = 0,

            NTablesCreated = 0, NTablesDeleted = 0, NTableFiles = 835, NCompacts = 0, NTTLCompacts = 0,

            NFileCountCompacts = 0, NWriterCompacts = 0, ActiveDiskUsage = 6840320, ActiveBloomFilterMemUsed = 2505,

            ActiveIndexBlocksSize = 35070, TotalDiskUsage = 6840320, TotalBloomFilterMemUsed = 2505,

---Type <return> to continue, or q <return> to quit---

            TotalIndexBlocksSize = 35070, TableObjectMemUsed = 207080, TableMetaMemUsed = 232965, MemtablesMemUsed = 1064960,

            BufferMemUsed = 0, ReadAheadBufferMemUsed = 0, NFailedCompacts = 0, TreeSnapshotMemoryUsed = 279408,

            PendingCheckpointBytes = 0, CheckpointOverhead = 0, WriteAmp = 0, ReadAmp = 0, ReadIOAmp = 0, BloomFilterFPR = 0,

            LogicalDataSize = 7535714, LogicalDiskSize = 7535714, Writer = {NInserts = 0, NBytesIncoming = 0,

              NWriteBytes = 0, NWriteIOs = 0, NSyncs = 0, NDataBlockWrites = 0, NIndexBlockWrites = 0}, Reader = {

              NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0}, CompactWriter = {

              NInserts = 0, NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0, NSyncs = 0, NDataBlockWrites = 0,

              NIndexBlockWrites = 0}, CompactReader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0,

              NIndexBlockReads = 0}, IterateReader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0,

              NIndexBlockReads = 0},

            LevelStats = {<std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >> = {

                _M_impl = {<std::allocator<magma::LSMTreeLevelStats>> = {<__gnu_cxx::new_allocator<magma::LSMTreeLevelStats>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >::_Vector_impl_data> = {_M_start = 0x7feeac701400, _M_finish = 0x7feeac7014f0,

                    _M_end_of_storage = 0x7feeac7014f0}, <No data fields>}}, <No data fields>}}, WalStats = {NTxns = 0,

            NInserts = 0, NWrites = 0, NWriteBytes = 0, NLoggedBytes = 0, NSegments = 3, NActiveSegments = 3,

            DiskUsed = 37867520, TxnSizeEstimate = 0, NSyncs = 0}}

        lsmStats = {UID = {static npos = 18446744073709551615,

            _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7feed7ee1980 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}},

          NItemsPut = 29512, NItemsDel = 0, NItemsGet = 14545, NBloomFilterHits = 23409, NBloomFilterMisses = 246,

          BytesIncoming = 30228468, BytesOutgoing = 0, NWriteBytes = 0, NReadBytes = 1038820, NSyncs = 0, NReadIOs = 455,

          NWriteIOs = 0, NItemsGarbageCollected = 0, NItemsMerged = 0, NItemsTransformed = 0, NItemsDropped = 0,

          NTablesCreated = 0, NTablesDeleted = 0, NTableFiles = 1899, NCompacts = 0, NTTLCompacts = 0,

          NFileCountCompacts = 0, NWriterCompacts = 0, ActiveDiskUsage = 515215360, ActiveBloomFilterMemUsed = 380411,

          ActiveIndexBlocksSize = 3681406, TotalDiskUsage = 768700416, TotalBloomFilterMemUsed = 380411,

          TotalIndexBlocksSize = 5498209, TableObjectMemUsed = 518816, TableMetaMemUsed = 605809,

          MemtablesMemUsed = 35946496, BufferMemUsed = 0, ReadAheadBufferMemUsed = 0, NFailedCompacts = 0,

          TreeSnapshotMemoryUsed = 943888, PendingCheckpointBytes = 0, CheckpointOverhead = 0, WriteAmp = 0, ReadAmp = 0,

          ReadIOAmp = 0.031282227569611548, BloomFilterFPR = 0.010399492707672797, LogicalDataSize = 517584773,

          LogicalDiskSize = 517584773, Writer = {NInserts = 0, NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0,

            NSyncs = 0, NDataBlockWrites = 0, NIndexBlockWrites = 0}, Reader = {NBytesOutgoing = 0, NReadBytes = 1038820,

---Type <return> to continue, or q <return> to quit---

            NReadIOs = 455, NDataBlockReads = 246, NIndexBlockReads = 246}, CompactWriter = {NInserts = 0,

            NBytesIncoming = 0, NWriteBytes = 0, NWriteIOs = 0, NSyncs = 0, NDataBlockWrites = 0, NIndexBlockWrites = 0},

          CompactReader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0},

          IterateReader = {NBytesOutgoing = 0, NReadBytes = 0, NReadIOs = 0, NDataBlockReads = 0, NIndexBlockReads = 0},

          LevelStats = {<std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >> = {

              _M_impl = {<std::allocator<magma::LSMTreeLevelStats>> = {<__gnu_cxx::new_allocator<magma::LSMTreeLevelStats>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<magma::LSMTreeLevelStats, std::allocator<magma::LSMTreeLevelStats> >::_Vector_impl_data> = {_M_start = 0x7feed407e000, _M_finish = 0x7feed407e190,

                  _M_end_of_storage = 0x7feed407e190}, <No data fields>}}, <No data fields>}}

        nKVStores = <optimized out>

        totalIndexSize = <optimized out>

        bfTotal = <optimized out>

#5  0x000000000085ccc0 in MagmaMemoryTrackingProxy::GetStats(std::chrono::duration<long, std::ratio<1l, 1000l> >) ()

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:207

        domainGuard = {previous = cb::Primary}

        stats = {_M_t = {<std::__uniq_ptr_impl<magma::Magma::MagmaStats, DomainAwareDelete<magma::Magma::MagmaStats> >> = {

              _M_t = {<std::_Tuple_impl<0, magma::Magma::MagmaStats*, DomainAwareDelete<magma::Magma::MagmaStats> >> = {<std::_Tuple_impl<1, DomainAwareDelete<magma::Magma::MagmaStats> >> = {<std::_Head_base<1, DomainAwareDelete<magma::Magma::MagmaStats>, true>> = {<DomainAwareDelete<magma::Magma::MagmaStats>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::Magma::MagmaStats*, false>> = {

                    _M_head_impl = 0x7fee3f25b000}, <No data fields>}, <No data fields>}}, <No data fields>}}

#6  0x000000000083d890 in MagmaKVStore::getAggrDbFileInfo (this=<optimized out>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:2936

        stats = {_M_t = {<std::__uniq_ptr_impl<magma::Magma::MagmaStats, DomainAwareDelete<magma::Magma::MagmaStats> >> = {

              _M_t = {<std::_Tuple_impl<0, magma::Magma::MagmaStats*, DomainAwareDelete<magma::Magma::MagmaStats> >> = {<std::_Tuple_impl<1, DomainAwareDelete<magma::Magma::MagmaStats> >> = {<std::_Head_base<1, DomainAwareDelete<magma::Magma::MagmaStats>, true>> = {<DomainAwareDelete<magma::Magma::MagmaStats>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::Magma::MagmaStats*, false>> = {

                    _M_head_impl = 0x7fee3f25b000}, <No data fields>}, <No data fields>}}, <No data fields>}}

        vbinfo = {fileSize = 0, spaceUsed = 0, prepareBytes = 0}

QE-Test:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.45831.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=false,GROUP=P0,upgrade_version=7.1.0-1316 -t storage.magma.magma_disk_full.MagmaDiskFull.test_crash_recovery_disk_full,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=1,GROUP=P0'

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Ankush Sharma

Reporter:: Ankush Sharma

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 21/Sep/21 10:16 AM

Updated:: 10/Oct/21 11:03 PM

Resolved:: 22/Sep/21 4:24 AM

Gerrit Reviews

There are no open Gerrit changes

Show There is 1 closed Gerrit change

Hide There is 1 closed Gerrit change

MB-48543 stats: Do no populate thread pool statistics in readonly mode: Gerrit Review:

[Magma] Memcached crashed in getQueue (workerType=magma::ThreadPool::Flusher) [DISK FULL TEST])

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty