Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48357

MagmaKVStore Magma open failed. Status:Corruption: Can't recover magma instance. KVStore-987 starting replay offset 671:2904064 < WAL starting offset 673:0

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.1.0
    • 7.1.0
    • storage-engine
    • 7.1.0-1250

    Description

      Steps:

      1. Step 1: Create a 10 node cluster
      2. Step 2: Create required buckets and collections.
      3. Step 1: Create 1000000 items sequentially
      4. Step 2: Create 1000000 items sequentially
      5. Step 2.1: Update 1000000 RandonKey keys to create 50 percent fragmentation
      6. Step 3: Create 1000000 items sequentially
      7. Step 3.1: Update 1000000 RandonKey keys to create 50 percent fragmentation
      8. Step 4: Rebalance in with Loading of docs followed by magma/memc crash 20 times on all nodes
      9. CRITICAL is seen on node .130 on 1st crash iteration in the step above

      172.23.121.130

      		 
      Found ' CRITICAL ' logs - ["[ns_server:info,2021-09-07T17:06:20.160-07:00,babysitter_of_ns_1@cb.local:<0.2257.2>:ns_port_server:log:221]memcached<0.2257.2>: 2021-09-07T17:06:19.959205-07:00 CRITICAL [(GleamBookUsers0) magma_3]Can't recover magma instance. KVStore-987 starting replay offset 671:2904064 < WAL starting offset 673:0\n", "memcached<0.2257.2>: 2021-09-07T17:06:19.959236-07:00 CRITICAL (GleamBookUsers0) MagmaKVStore Magma open failed. Status:Corruption: Can't recover magma instance. KVStore-987 starting replay offset 671:2904064 < WAL starting offset 673:0\n"]

      QE Test

      		 
      git fetch "http://review.couchbase.org/TAF" refs/changes/34/151634/125 && git checkout FETCH_HEAD
      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=10,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=20,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=50000,doc_ops=create:update:delete:read,durability=Majority,crashes=10 -m rest'

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-48707 [Magma] - Minidumps seen during graceful failover + rebalance out + CRUD on collections

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apaar.gupta Apaar Gupta
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty