Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49019

[Magma, KV+XDCR, SmallScale]: XDCR replication seems to be stuck.

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.0
    • 7.1.0
    • XDCR
    • 7.1.0-1521

    Description

      Steps:

      1. Create a 9 node cluster
      2. Create a 5 node XDCR remote cluster
      3. Create required buckets and 10 collections.
      4. Create required buckets and 10 collections on XDCR remote.
      5. Create 1000000 items/collection with durability majority
      6. Update 1000000 keys to create 50 percent fragmentation
      7. Create new 1000000 items/collection with durability majority
      8. Update new 1000000 keys to create 50 percent fragmentation
      9. Start a CRUD data load asynchronously.
      10. Rebalance in with Loading of docs. Crash and resume rebalance at 20%, 40%, 60%, 80%.
      11. Crash Magma/memc on source cluster 20 times with Loading of docs
      12. Rebalance Out with Loading of docs. Crash and resume rebalance at 20%, 40%, 60%, 80%.
      13. XDCR replication seems to be stuck. Since 30 mins 64.005M mutations remaining
      14. Also, because there were continuous CRUD going on the in the source cluster items in destination cluster are more than in source. Not sure if this is expected?

      QE Test

      		 
      git fetch "http://review.couchbase.org/TAF" refs/changes/06/163706/1 && git checkout FETCH_HEAD
      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=10,xdcr_collections=10,maxttl=10,num_indexes=20,pc=10,xdcr_remote_nodes=5,index_nodes=0,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=50000,doc_ops=create:update:delete:read,durability=Majority,crashes=10 -m rest'

      I have this observation with XDCR that the RAM usage on the source/Dstn cluster is always very high while there is no mutation on the source cluster and just XDCR is runing.
      Source

      Destination

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-48674 XDCR - P2P - bg process to pull ckpts under error conditions

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-49024 XDCR - P2P - backfill pipeline not honoring dynamic wait time

          • Major - Major loss of function.
          • Closed
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-49030 primary/secondary memory domain stats incorrect

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            View 5 older comments
            neil.huang Neil Huang added a comment -

            Main pipeline seems to be fine:

            FILE collectinfo-2021-10-19t180046-ns_1@172.23.120.170.zip =======================
            		 
             
            		 
            cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:55.832-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=0, docs_processed=0, changes_left=0
            		 
            cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:57.213-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8926006, docs_processed=0, changes_left=8926006
            		 
            cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:58.095-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8927222, docs_processed=0, changes_left=8927222
            		 
            cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:59.089-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8927225, docs_processed=0, changes_left=8927225
            		 
            cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:09:00.091-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8930428, docs_processed=0, changes_left=8930428
            		 
            …
            		 
            cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T11:09:36.113-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14846146, docs_processed=14846146, changes_left=0
            		 
             
            		 
             
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.116.zip =======================
            		 
            cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T02:09:19.273-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=0, docs_processed=0, changes_left=0
            		 
            cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T02:09:20.925-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=77994, docs_processed=21511, changes_left=56483
            		 
            cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T02:09:21.924-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=78010, docs_processed=25512, changes_left=52498
            		 
            …
            		 
             
            		 
            cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T11:05:10.953-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14977946, docs_processed=14977946, changes_left=0
            		 
            cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T11:05:11.953-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14977946, docs_processed=14977946, changes_left=0
            		 
             
            		 
             
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.124.zip =======================
            		 
            cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:08:56.087-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=0, docs_processed=0, changes_left=0
            		 
            cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:08:57.261-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8922547, docs_processed=0, changes_left=8922547
            		 
            cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:08:59.027-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8923428, docs_processed=0, changes_left=8923428
            		 
            cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:09:00.033-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8926181, docs_processed=0, changes_left=8926181
            		 
             
            		 
            …
            		 
             
            		 
            cbcollect_info_ns_1@172.23.121.126_20211019-180050/ns_server.goxdcr.log:2021-10-19T11:08:29.661-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14896920, docs_processed=14896920, changes_left=0

            I would say that the number of mutations you're seeing is probably due to the inability to start backfill pipelines due to a few issues. I'll link them in this MB.

            neil.huang Neil Huang added a comment - Main pipeline seems to be fine: FILE collectinfo-2021-10-19t180046-ns_1@172.23.120.170.zip =======================   cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:55.832-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=0, docs_processed=0, changes_left=0 cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:57.213-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8926006, docs_processed=0, changes_left=8926006 cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:58.095-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8927222, docs_processed=0, changes_left=8927222 cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:08:59.089-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8927225, docs_processed=0, changes_left=8927225 cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T02:09:00.091-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8930428, docs_processed=0, changes_left=8930428 … cbcollect_info_ns_1@172.23.120.170_20211019-180049/ns_server.goxdcr.log:2021-10-19T11:09:36.113-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14846146, docs_processed=14846146, changes_left=0     FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.116.zip ======================= cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T02:09:19.273-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=0, docs_processed=0, changes_left=0 cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T02:09:20.925-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=77994, docs_processed=21511, changes_left=56483 cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T02:09:21.924-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=78010, docs_processed=25512, changes_left=52498 …   cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T11:05:10.953-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14977946, docs_processed=14977946, changes_left=0 cbcollect_info_ns_1@172.23.121.116_20211019-180047/ns_server.goxdcr.log:2021-10-19T11:05:11.953-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14977946, docs_processed=14977946, changes_left=0     FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.124.zip ======================= cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:08:56.087-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=0, docs_processed=0, changes_left=0 cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:08:57.261-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8922547, docs_processed=0, changes_left=8922547 cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:08:59.027-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8923428, docs_processed=0, changes_left=8923428 cbcollect_info_ns_1@172.23.121.124_20211019-180050/ns_server.goxdcr.log:2021-10-19T02:09:00.033-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=8926181, docs_processed=0, changes_left=8926181   …   cbcollect_info_ns_1@172.23.121.126_20211019-180050/ns_server.goxdcr.log:2021-10-19T11:08:29.661-07:00 INFO GOXDCR.StatsMgr: e36f78aae37740197f0fd990ee484227/GleamBookUsers0/GleamBookUsers0 total_docs=14896920, docs_processed=14896920, changes_left=0 I would say that the number of mutations you're seeing is probably due to the inability to start backfill pipelines due to a few issues. I'll link them in this MB.
            neil.huang Neil Huang added a comment -

            In the meantime if you turn off P2P you should be able to get unblocked

            neil.huang Neil Huang added a comment - In the meantime if you turn off P2P you should be able to get unblocked
            neil.huang Neil Huang added a comment -

            Memory wise, here's the usage. The max is 1600290816 by node 129 which is about 1.49 GB.

            zipgrepCbCollectXdcrTopGoxdcrMemory | grep -v FILE | grep -v Clone | grep -v heap | grep -v for | awk '{print $2}' | sort -n
            		 
            453296128
            		 
            736157696
            		 
            741539840
            		 
            857554944
            		 
            879681536
            		 
            1178091520
            		 
            1485471744
            		 
            1508794368
            		 
            1600290816

            Trace wise, this is being worked on as MB-48889

            FILE collectinfo-2021-10-19t180046-ns_1@172.23.120.170.zip =======================
            		 
            heap profile: 2967: 944190816 [407706: 25814793088] @ heap/1048576
            		 
            1471: 879681536 [21745: 13003857920] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.116.zip =======================
            		 
            heap profile: 2967: 520960568 [616067: 28055843800] @ heap/1048576
            		 
            758: 453296128 [20648: 12347834368] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.124.zip =======================
            		 
            heap profile: 4994: 1684465296 [590896: 29239874112] @ heap/1048576
            		 
            2484: 1485471744 [20913: 12506308608] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.126.zip =======================
            		 
            heap profile: 2628: 789272976 [328979: 23836952672] @ heap/1048576
            		 
            1231: 736157696 [21617: 12927311872] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.127.zip =======================
            		 
            heap profile: 4932: 1697172864 [512581: 29020569448] @ heap/1048576
            		 
            2523: 1508794368 [21320: 12749701120] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.128.zip =======================
            		 
            heap profile: 3203: 938742712 [427921: 25978389472] @ heap/1048576
            		 
            1434: 857554944 [21751: 13007446016] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.129.zip =======================
            		 
            heap profile: 4975: 1863978192 [427660: 25546669032] @ heap/1048576
            		 
            2676: 1600290816 [21343: 12763455488] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.130.zip =======================
            		 
            heap profile: 2789: 811993104 [503844: 27821964288] @ heap/1048576
            		 
            1240: 741539840 [21826: 13052297216] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            		 
            FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.48.zip =======================
            		 
            heap profile: 4736: 1441074616 [525424: 28881340904] @ heap/1048576
            		 
            1970: 1178091520 [21161: 12654616576] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1
            		 
            #	0x806cec	github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec				/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523

            neil.huang Neil Huang added a comment - Memory wise, here's the usage. The max is 1600290816 by node 129 which is about 1.49 GB. zipgrepCbCollectXdcrTopGoxdcrMemory | grep -v FILE | grep -v Clone | grep -v heap | grep -v for | awk '{print $2}' | sort -n 453296128 736157696 741539840 857554944 879681536 1178091520 1485471744 1508794368 1600290816 Trace wise, this is being worked on as MB-48889 FILE collectinfo-2021-10-19t180046-ns_1@172.23.120.170.zip ======================= heap profile: 2967: 944190816 [407706: 25814793088] @ heap/1048576 1471: 879681536 [21745: 13003857920] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.116.zip ======================= heap profile: 2967: 520960568 [616067: 28055843800] @ heap/1048576 758: 453296128 [20648: 12347834368] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.124.zip ======================= heap profile: 4994: 1684465296 [590896: 29239874112] @ heap/1048576 2484: 1485471744 [20913: 12506308608] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.126.zip ======================= heap profile: 2628: 789272976 [328979: 23836952672] @ heap/1048576 1231: 736157696 [21617: 12927311872] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.127.zip ======================= heap profile: 4932: 1697172864 [512581: 29020569448] @ heap/1048576 2523: 1508794368 [21320: 12749701120] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.128.zip ======================= heap profile: 3203: 938742712 [427921: 25978389472] @ heap/1048576 1434: 857554944 [21751: 13007446016] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.129.zip ======================= heap profile: 4975: 1863978192 [427660: 25546669032] @ heap/1048576 2676: 1600290816 [21343: 12763455488] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.130.zip ======================= heap profile: 2789: 811993104 [503844: 27821964288] @ heap/1048576 1240: 741539840 [21826: 13052297216] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523 FILE collectinfo-2021-10-19t180046-ns_1@172.23.121.48.zip ======================= heap profile: 4736: 1441074616 [525424: 28881340904] @ heap/1048576 1970: 1178091520 [21161: 12654616576] @ 0x411f79 0x413c29 0x41720a 0x806ced 0xa9affa 0xa975a7 0xac535e 0x473ee1 # 0x806cec github.com/couchbase/goxdcr/base.DcpStatsMapType.Clone+0xec /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/base/types.go:2523
            neil.huang Neil Huang added a comment -

            https://issues.couchbase.com/browse/MB-49024 resolved will help the situation where backfill pipeline couldn't start and keep restarting.
            You can always try to turn off p2p like I mentioned below.

            neil.huang Neil Huang added a comment - https://issues.couchbase.com/browse/MB-49024 resolved will help the situation where backfill pipeline couldn't start and keep restarting. You can always try to turn off p2p like I mentioned below.
            ritesh.agarwal Ritesh Agarwal added a comment - - edited

            It is working fine without turning p2p off on the recent volume runs. Last one is on 2305 build where replication completed properly at 4TB src : 400M dstn cluster.

            ritesh.agarwal Ritesh Agarwal added a comment - - edited It is working fine without turning p2p off on the recent volume runs. Last one is on 2305 build where replication completed properly at 4TB src : 400M dstn cluster.

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty