Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48559

[30TB test]: XDCR replications are stuck at 11.116G mutations remaining since hours. Total Data at source is 28TB(10B items).

    XMLWordPrintable

Details

    Description

      1. Create a src cluster with 3 nodes, Magma bucket, 50 collections, 10B items, 50% fragmentation.
      2. While the above load was in progress. Create a XDCR dstn cluster at .70. That time there were around 7-8B items in src cluster.
      3. Create replication on 5 collections. Each having 260GB active data.
      4. Replication is now progressing.
      5. There are a bunch of XDCR error on the src cluster side on 172.23.110.65, 172.23.110.66:

      2021-09-22 06:29:14 172.23.110.66:genericPipeline.vbMasterCheckFunc:172.23.110.64:8091 : Execution timed out - did not hear back from node after 51s. Could be due to peer node being busy to respond in time or this XDCR being too busy to handle incoming requests
       
      2021-09-22 06:27:22 172.23.110.66:genericPipeline.mergeCkptFunc:There is split decision on internalSpecIDs
      

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'
      

      Thanks to Pavithra Mahamani for adding the xdcr replication to the src cluster!

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty