Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.1.0
-
7.1.0-1250
-
Untriaged
-
-
1
-
Unknown
-
Magma 2021-Sep-6, Magma May 2022
Description
Steps:
- Step 1: Create a 10 node cluster
- Step 2: Create required buckets and collections.
- Step 1: Create 1000000 items sequentially
- Step 2: Create 1000000 items sequentially
- Step 2.1: Update 1000000 RandonKey keys to create 50 percent fragmentation
- Step 3: Create 1000000 items sequentially
- Step 3.1: Update 1000000 RandonKey keys to create 50 percent fragmentation
- Step 4: Rebalance in with Loading of docs followed by magma/memc crash 20 times on all nodes
- CRITICAL is seen on node .130 on 1st crash iteration in the step above
172.23.121.130 |
Found ' CRITICAL ' logs - ["[ns_server:info,2021-09-07T17:06:20.160-07:00,babysitter_of_ns_1@cb.local:<0.2257.2>:ns_port_server:log:221]memcached<0.2257.2>: 2021-09-07T17:06:19.959205-07:00 CRITICAL [(GleamBookUsers0) magma_3]Can't recover magma instance. KVStore-987 starting replay offset 671:2904064 < WAL starting offset 673:0\n", "memcached<0.2257.2>: 2021-09-07T17:06:19.959236-07:00 CRITICAL (GleamBookUsers0) MagmaKVStore Magma open failed. Status:Corruption: Can't recover magma instance. KVStore-987 starting replay offset 671:2904064 < WAL starting offset 673:0\n"]
|
QE Test |
git fetch "http://review.couchbase.org/TAF" refs/changes/34/151634/125 && git checkout FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=10,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=20,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=50000,doc_ops=create:update:delete:read,durability=Majority,crashes=10 -m rest'
|
Attachments
Issue Links
- is caused by
-
MB-48707 [Magma] - Minidumps seen during graceful failover + rebalance out + CRUD on collections
- Closed