Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46099

130 node testing: Rebalance in 1 node in a 85 node cluster failed due to buckets_cleanup_failed

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.2
    • ns_server
    • 7.0.0-5085

    Description

      Steps:
      1. Create a 85 node cluster
      2. Create a bucket and 20 collections
      3. Start load: maxttl=60, durability=Majority
      4. Rebalance in 172.23.96.89. Rebalance failed due to bas replicas. buckets_cleanup_failed

      QE test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SystemTestMagma,nodes_init=85,replicas=1,skip_cleanup=True,num_items=1000000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,compression_mode=off,iterations=100,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,skip_cleanup=True,key_size=18,randomize_doc_size=True,randomize_value=True,assert_crashes_on_load=True,doc_ops=update,maxttl=60,num_collections=20,durability=Majority,fragmentation=20,pc=2,crashes=10,sdk_client_pool=True -m rest'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            ritesh.agarwal Ritesh Agarwal created issue -
            ritesh.agarwal Ritesh Agarwal made changes -
            Field Original Value New Value
            Link to Log File, atop/blg, CBCollectInfo, Core dump http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::0

            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.95.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.99.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.96.89.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.163.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.164.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.53.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.56.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.62.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.120.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.125.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.170.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.123.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.127.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.128.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.129.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.131.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.135.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.48.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.208.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.240.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.241.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.242.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.243.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.245.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.247.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.248.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.249.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.10.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.11.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.12.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.4.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.5.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.6.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.7.zip
            http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::0
            http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::1


            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.95.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.99.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.96.89.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.163.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.164.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.53.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.56.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.62.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.120.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.125.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.170.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.123.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.127.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.128.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.129.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.131.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.135.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.48.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.208.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.240.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.241.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.242.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.243.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.245.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.247.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.248.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.249.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.10.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.11.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.12.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.4.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.5.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.6.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.7.zip
            ritesh.agarwal Ritesh Agarwal made changes -
            Summary 130 node testing: Rebalance in 1 node in a 85 node cluster leads to bad replicators port rebalance 130 node testing: Rebalance in 1 node in a 85 node cluster leads to bad replicators post rebalance
            ritesh.agarwal Ritesh Agarwal made changes -
            Link to Log File, atop/blg, CBCollectInfo, Core dump http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::0
            http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::1


            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.95.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.99.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.96.89.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.163.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.164.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.53.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.56.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.62.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.120.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.125.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.170.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.123.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.127.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.128.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.129.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.131.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.135.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.48.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.208.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.240.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.241.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.242.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.243.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.245.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.247.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.248.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.249.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.10.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.11.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.12.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.4.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.5.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.6.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.7.zip
            http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::0
            http://supportal.couchbase.com/snapshot/28b5d7a41e42afa4b17f82f6d79bdfde::1


            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.95.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.99.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.96.89.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.163.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.164.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.53.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.56.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.62.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.120.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.125.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.170.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.123.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.124.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.127.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.128.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.129.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.131.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.135.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.48.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.208.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.240.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.241.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.242.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.243.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.245.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.247.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.248.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.249.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.10.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.11.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.12.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.4.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.5.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.6.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.123.7.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.155.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.106.159.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.1.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.110.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.112.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.12.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.16.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.18.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.31.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.32.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.107.33.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.119.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.122.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.123.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.126.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.127.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.128.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.120.129.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.115.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.116.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.126.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.130.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.132.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.133.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.134.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.136.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.139.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.140.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.141.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.142.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.143.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.144.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.152.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.153.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.154.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.155.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.156.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.157.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.162.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.76.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.121.79.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.221.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.223.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.224.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.225.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.226.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.227.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.228.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.229.zip
            s3://cb-customers-secure/bad_replicas/2021-05-04/collectinfo-2021-05-04t121859-ns_1@172.23.122.231.zip
            ritesh.agarwal Ritesh Agarwal made changes -
            Summary 130 node testing: Rebalance in 1 node in a 85 node cluster leads to bad replicators post rebalance 130 node testing: Rebalance in 1 node in a 85 node cluster leads to bad replicators and rebalance failed.
            owend Daniel Owen made changes -
            Assignee Dave Finlay [ dfinlay ] Daniel Owen [ owend ]
            sumedh.basarkod Sumedh Basarkod (Inactive) made changes -
            Environment 5.0.0-5085 7.0.0-5085
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ritesh Agarwal [ ritesh.agarwal ]
            ritesh.agarwal Ritesh Agarwal made changes -
            Description Steps:
            1. Create a 85 node cluster
            2. Create a bucket and 20 collections
            3. Start load: maxttl=60, durability=Majority
            4. Rebalance in 172.23.96.89. Rebalance failed due to bas replicas. buckets_cleanup_failed
            Steps:
            1. Create a 85 node cluster
            2. Create a bucket and 20 collections
            3. Start load: maxttl=60, durability=Majority
            4. Rebalance in 172.23.96.89. Rebalance failed due to bas replicas. buckets_cleanup_failed

            {code:title=QE test}
            guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SystemTestMagma,nodes_init=85,replicas=1,skip_cleanup=True,num_items=1000000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,compression_mode=off,iterations=100,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,skip_cleanup=True,key_size=18,randomize_doc_size=True,randomize_value=True,assert_crashes_on_load=True,doc_ops=update,maxttl=60,num_collections=20,durability=Majority,fragmentation=20,pc=2,crashes=10,sdk_client_pool=True -m rest'
            {code}
            ritesh.agarwal Ritesh Agarwal made changes -
            Assignee Ritesh Agarwal [ ritesh.agarwal ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Summary 130 node testing: Rebalance in 1 node in a 85 node cluster leads to bad replicators and rebalance failed. 130 node testing: Rebalance in 1 node in a 85 node cluster failed due to buckets_cleanup_failed
            owend Daniel Owen made changes -
            Component/s couchbase-bucket [ 10173 ]
            Component/s ns_server [ 10019 ]
            owend Daniel Owen made changes -
            Sprint KV-Engine CC Final Sprint [ 1548 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ritam.sharma Ritam Sharma made changes -
            Labels 130nodetest 130nodetest affects-cc-testing
            owend Daniel Owen made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            owend Daniel Owen made changes -
            Link This issue duplicates MB-45594 [ MB-45594 ]
            owend Daniel Owen made changes -
            Component/s ns_server [ 10019 ]
            Component/s couchbase-bucket [ 10173 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Meni Hillel [ JIRAUSER25407 ]
            dfinlay Dave Finlay made changes -
            Link This issue is duplicated by MB-46067 [ MB-46067 ]
            dfinlay Dave Finlay made changes -
            Link This issue is duplicated by MB-45594 [ MB-45594 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Assignee Meni Hillel [ JIRAUSER25407 ] Aliaksey Artamonau [ aliaksey artamonau ]
            lynn.straus Lynn Straus made changes -
            Due Date 25/May/21
            lynn.straus Lynn Straus made changes -
            Due Date 25/May/21 21/May/21
            lynn.straus Lynn Straus made changes -
            Due Date 21/May/21 24/May/21
            lynn.straus Lynn Straus made changes -
            Link This issue blocks MB-44894 [ MB-44894 ]
            lynn.straus Lynn Straus made changes -
            Labels 130nodetest affects-cc-testing 130nodetest affects-cc-testing approved-for-7.0.0
            dfinlay Dave Finlay made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Fix Version/s Cheshire-Cat [ 15915 ]
            lynn.straus Lynn Straus made changes -
            Labels 130nodetest affects-cc-testing approved-for-7.0.0 130nodetest approved-for-7.0.0
            lynn.straus Lynn Straus made changes -
            Labels 130nodetest approved-for-7.0.0 130nodetest
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.1.0 [ 17117 ]
            Fix Version/s 7.0.1 [ 17104 ]
            lynn.straus Lynn Straus made changes -
            Link This issue blocks MB-44894 [ MB-44894 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s Neo [ 17615 ]
            Fix Version/s 7.1.0 [ 17117 ]
            wayne Wayne Siu made changes -
            Link This issue relates to MB-47169 [ MB-47169 ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Fix Version/s Neo [ 17615 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Assignee Aliaksey Artamonau [ aliaksey artamonau ] Ritesh Agarwal [ ritesh.agarwal ]
            ritesh.agarwal Ritesh Agarwal made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty