Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49702

Magma rolling back to zero despite rollback seqno being within the 10min/10% window

    XMLWordPrintable

Details

    Description

      1. Step 1: Create a 10 node cluster
      2. Step 2: Create required buckets and collections.
      3. Step 3: Create 1000000 items sequentially
      4. Step 4: Update 1000000 RandonKey keys to create 50 percent fragmentation
      5. Step 5: Create 1000000 items sequentially
      6. Step 6: Update 1000000 RandonKey keys to create 50 percent fragmentation
      7. Step 7: Add 5 index/n1ql nodes and create 20 indexes on 20 different collections. Let them build asynchoronously.
      8. Step 8: Start another CRUD load asynchoronously.
      9. Step 9: Rebalance in with Loading of docs.

      Rebalance exited with reason {mover_crashed,
      		 
      {unexpected_exit,
      		 
      {'EXIT',<0.14702.117>,
      		 
      {{dcp_wait_for_data_move_failed,
      		 
      "GleamBookUsers0",378,'ns_1@172.23.121.123',
      		 
      ['ns_1@172.23.121.127',
      		 
      'ns_1@172.23.121.136'],
      		 
      {error,no_stats_for_this_vbucket}},
      		 
      [{ns_single_vbucket_mover,
      		 
      '-wait_dcp_data_move/5-fun-0-',5,
      		 
      [{file,"src/ns_single_vbucket_mover.erl"},
      		 
      {line,483}]},
      		 
      {proc_lib,init_p,3,
      		 
      [{file,"proc_lib.erl"},{line,211}]}]}}}}.
      		 
      Rebalance Operation Id = 8ebe25aa8db470fbc48b47fc2bf9e0b3

      QE Test

      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=10,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=50000,doc_ops=create:update:delete:read,durability=Majority,crashes=10 -m rest'

      Now re-running the test with just 5 indexes to see if that behaves differently.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-43796 OSO Enabled Initial Index build time greater than OSO Disabled

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-49786 Default number of AuxIO threads too low for HiDD workloads

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-49685 Seqno-ordered collection filtered backfills monopolise task/thread

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MB-49006 Prioritise backfills for replication DCP streams

          • Major - Major loss of function.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty