Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
7.1.0-1807
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
-
KV 2021-Dec
Description
Steps to repro:
- Create a 3 node cluster(172.23.121.139, 172.23.121.140, 172.23.121.141(bucket_ram_quota = 2056 MB/node), replicas=1)
- Create 100 non default collections in default scope(collection count is 101(including default collection)
- Start Load 300k docs in each of 100 non default collections
- Trigger full compaction
- Create two scopes(scope1 and scope2) and 5 collections in each of these scopes
- While data loading is going on remove node 172.23.121.141 and change replica to 0
- Triggered Rebalance
- Drop one scope (scope2) and two collections in scope1
- Rebalance was successful
- Again change replica to 1 and rebalance
- Recreate a new scope (scope2) and two collections in scope1
- Drop the scope2
- Observed Memcached aborted in NexusKVStoreSecondaryGetAllKeysCallback::callback(DiskDocKey const&) ()
Core found on node : 172.23.121.140
BackTrace:
(gdb) bt full
|
#0 0x00007fdc1420d387 in raise () from /lib64/libc.so.6
|
No symbol table info available.
|
#1 0x00007fdc1420ea78 in abort () from /lib64/libc.so.6
|
No symbol table info available.
|
#2 0x0000000000497316 in cb::handleError(spdlog::logger&, spdlog::level::level_enum, std::basic_string_view<char, std::char_traits<char> >, cb::ErrorHandlingMethod) [clone .cold] ()
|
No symbol table info available.
|
#3 0x0000000000902769 in NexusKVStoreSecondaryGetAllKeysCallback::callback(DiskDocKey const&) ()
|
No symbol table info available.
|
#4 0x000000000086e8e1 in MagmaKVStore::getAllKeys(Vbid, DiskDocKey const&, unsigned int, std::shared_ptr<StatusCallback<DiskDocKey const&> >) const::{lambda(magma::Slice&, magma::Slice&, magma::Slice&)#1}::operator()(magma::Slice&, magma::Slice&, magma::Slice&) const [clone .constprop.0] ()
|
No symbol table info available.
|
#5 0x000000000088f216 in std::_Function_handler<void (magma::Slice&, magma::Slice&, magma::Slice&), MagmaMemoryTrackingProxy::GetRange(unsigned short, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long)::{lambda(magma::Slice&, magma::Slice&, magma::Slice&)#1}>::_M_invoke(std::_Any_data const&, magma::Slice&, magma::Slice&, magma::Slice&) ()
|
No symbol table info available.
|
#6 0x000000000098557e in magma::KVStore::IterateByKey(std::shared_ptr<magma::KVStoreSnapshot>&, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
|
No symbol table info available.
|
#7 0x00000000009284c5 in magma::Magma::GetRange(magma::Magma::Snapshot&, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
|
No symbol table info available.
|
#8 0x000000000092ab62 in magma::Magma::GetRange(unsigned short, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
|
No symbol table info available.
|
#9 0x00000000008902c2 in MagmaMemoryTrackingProxy::GetRange(unsigned short, magma::Slice const&, magma::Slice const&, std::function<void (magma::Slice&, magma::Slice&, magma::Slice&)>, bool, unsigned long) ()
|
No symbol table info available.
|
#10 0x0000000000875867 in MagmaKVStore::getAllKeys(Vbid, DiskDocKey const&, unsigned int, std::shared_ptr<StatusCallback<DiskDocKey const&> >) const
|
()
|
No symbol table info available.
|
#11 0x00000000008ed3f2 in NexusKVStore::getAllKeys(Vbid, DiskDocKey const&, unsigned int, std::shared_ptr<StatusCallback<DiskDocKey const&> >) const
|
()
|
No symbol table info available.
|
#12 0x00000000006d8e23 in FetchAllKeysTask::run() ()
|
QE-TEST:
git fetch "https://review.couchbase.org/TAF" refs/changes/88/166488/2 && git checkout FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/qe_r.ini -p bucket_storage=couchstore,rerun=false,bucket_eviction_policy=fullEviction,init_loading=False -t storage.magma.magma_rebalance.MagmaRebalance.test_data_load_collections_with_rebalance_in,num_items=300000,doc_size=256,nodes_init=3,nodes_in=1,standard_buckets=1,magma_buckets=0,bucket_storage=couchstore,data_load_stage=during,sdk_timeout=60,vbuckets=1024,key_size=12,replicas=1,infra_log_level=debug,log_level=debug,skip_cleaup=True,randomize_value=True,bucket_eviction_policy=fullEviction,infra_log_level=debug,log_level=debug,init_loading=False,fragmentation=50,skip_cleanup=True,autoCompactionDefined=true,iterations=1,enable_dp=True,num_collections=101,num_scopes=1,bucket_ram_quota=2056,skip_cleanup=True,sdk_client_pool=False,ops_rate=30000,doc_ops=create,create_perc=0,delete_perc=50,update_perc=50,num_collections_to_drop=50,get-cbcollect-info=True -m rest'
|
Attachments
Issue Links
- relates to
-
MB-49390 NexusKVStore automated A/B verification of KVStore implementations
-
- Closed
-
| For Gerrit Dashboard: MB-49839 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 166911,4 | MB-49839: Remove unused variable | master | kv_engine | Status: MERGED | +2 | +1 |
| 166912,3 | MB-49839: Return early in CouchKVStore::getAllKeys | master | kv_engine | Status: MERGED | +2 | +1 |
| 166965,7 | MB-49839 magma: Add bool return to GetRange callback function | master | magma | Status: MERGED | +2 | +1 |
| 167009,7 | MB-49839: Adapt to GetRange API change | master | kv_engine | Status: MERGED | +2 | +1 |
| 167301,3 | MB-49839: [Magma] Cancel getAllKeys in same way that CouchKVStore does | master | kv_engine | Status: MERGED | +2 | +1 |