Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.0
-
7.2.0-5177
-
Untriaged
-
Centos 64-bit
-
-
0
-
Unknown
Description
Steps To Recreate:
- Create a 3 node cluster
- Create a magma buckets (history params not set) (Vbuckets=1024, replicas=2)
- Create fifty collections(total collections including default are fifty one)
- Create 100K docs in each of the collections
- Update 100K docs in each of the collections while update is going on disable replicas and rebalance
- Update bucket params with history_retention_seconds=86400, history_retention_bytes=96000000000
- Start upserting the docs again
- Observed Memcached crashes in fail_fast_throw ( message=0xcac1f8 "GSL: Precondition failure: 'lsmStats.HistoryLogicalDiskSize >= historyLogicalGarbageSize'
Below Core Dump is on node 172.23.107.239
BackTrace:
(gdb) bt full
|
#0 0x00007f5edcb58387 in raise () from /lib64/libc.so.6
|
No symbol table info available.
|
#1 0x00007f5edcb59a78 in abort () from /lib64/libc.so.6
|
No symbol table info available.
|
#2 0x00007f5edd4a363c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
|
terminating = false
|
t = <optimized out>
|
#3 0x0000000000b4273b in backtrace_terminate_handler() ()
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
|
No locals.
|
#4 0x00007f5edd4ae8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
|
No locals.
|
#5 0x00007f5edd4ae961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
|
No locals.
|
#6 0x00007f5edd4aebf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f5ed4004ba0, tinfo=tinfo@entry=0xc54a08 <typeinfo for gsl::fail_fast>,
|
dest=dest@entry=0x598ea0 <gsl::fail_fast::~fail_fast()>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
|
globals = <optimized out>
|
header = 0x7f5ed4004b20
|
#7 0x000000000050f05c in fail_fast_throw (
|
message=0xcac1f8 "GSL: Precondition failure: 'lsmStats.HistoryLogicalDiskSize >= historyLogicalGarbageSize' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:139")
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/gsl-lite.hpp:1769
|
No locals.
|
#8 magma::LSMTree::GetStats (this=0x7f5e24887010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:139
|
dataLevel = <optimized out>
|
logicalGarbageSize = <optimized out>
|
historyLogicalGarbageSize = <optimized out>
|
lsmStats = {UID = {static npos = 18446744073709551615,
|
_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
|
_M_p = 0x7f5edb5f5c40 <Address 0x7f5edb5f5c40 out of bounds>}, _M_string_length = 52, {
|
_M_local_buf = "4\000\000\000\000\000\000\000=\232}\336^\177\000", _M_allocated_capacity = 52}}, NItemsPut = 15068, NItemsDel = 0,
|
NItemsGet = 0, BytesIncoming = 9226086, BytesOutgoing = 0, NWriteBytes = 19359790, NReadBytes = 6875895, NSyncs = 316, NReadIOs = 3305,
|
NWriteIOs = 1433, NItemsGarbageCollected = 4344, NItemsMerged = 4344, NItemsTransformed = 0, NItemsDropped = 0, NTablesCreated = 1112,
|
NTablesDeleted = 538, NTableFiles = 34, NCompacts = 179, NTTLCompacts = 0, NFileCountCompacts = 21, NNonL0LocalCompacts = 0,
|
QE-TEST:
git fetch https://review.couchbase.org/TAF refs/changes/62/186462/22 && git checkout -b change-186462 FETCH_HEAD
|
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol.ini -p bucket_storage=magma,bucket_ram_quota=256,init_loading=False,bucket_eviction_policy=fullEviction,rerun=False -t storage.magma.cdc_steady_state.SteadyStateTests.test_history_retention_for_n_update_iterations,nodes_init=3,skip_cleanup=True,num_items=100000,doc_size=1024,batch_size=100,sdk_timeout=60,log_level=info,infra_log_level=info,key_size=22,num_collections=51,ops_rate=20000,key_type=RandomKey,vbuckets=1024,replicas=2,test_itr=50,bucket_history_retention_seconds=0,bucket_history_retention_bytes=0,standard_buckets=1,magma_buckets=1,num_scopes=1 -m rest'
|
Attachments
For Gerrit Dashboard: MB-55609 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
186850,4 | MB-55609 magma: Fix incorrect expects on garbage size | neo | magma | Status: MERGED | +2 | +1 |