Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48568

[Magma, 30TB, 1%]: Rebalance exited with reason mover_crashed, dcp_wait_for_data_move_failed after being 0% progress for 4 hours

    XMLWordPrintable

Details

    Description

      Note: Did not hit this on 7.1.0-1247

      1. Create a 3 kv node and 1 index/n1ql node cluster
      2. Create magma bucket, 50 collections under default scope
      3. Load 5B items and upsert them
      4. Load another 5B items and upsert them as well
      5. Create 50 indexes on 50 collections and build them. Start 50 QPS
      6. Rebalance In 1 node. Rebalance failed

      172.23.110.64

      		 
      Rebalance exited with reason {mover_crashed,
      		 
      {unexpected_exit,
      		 
      {'EXIT',<0.15895.257>,
      		 
      {{dcp_wait_for_data_move_failed,
      		 
      "GleamBookUsers0",1023,'ns_1@172.23.110.66',
      		 
      ['ns_1@172.23.110.69'],
      		 
      {error,no_stats_for_this_vbucket}},
      		 
      [{ns_single_vbucket_mover,
      		 
      '-wait_dcp_data_move/5-fun-0-',5,
      		 
      [{file,"src/ns_single_vbucket_mover.erl"},
      		 
      {line,483}]},
      		 
      {proc_lib,init_p,3,
      		 
      [{file,"proc_lib.erl"},{line,234}]}]}}}}.
      		 
      Rebalance Operation Id = 845763b4fe63318184db4ee413c6246f

      QE Test

      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-48473 MagmaKVStore incrementing purgeSeqno when purging complete prepares and dropping collection items during implicit compaction

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty