Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46099

130 node testing: Rebalance in 1 node in a 85 node cluster failed due to buckets_cleanup_failed

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.2
    • ns_server
    • 7.0.0-5085

    Description

      Steps:
      1. Create a 85 node cluster
      2. Create a bucket and 20 collections
      3. Start load: maxttl=60, durability=Majority
      4. Rebalance in 172.23.96.89. Rebalance failed due to bas replicas. buckets_cleanup_failed

      QE test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SystemTestMagma,nodes_init=85,replicas=1,skip_cleanup=True,num_items=1000000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,compression_mode=off,iterations=100,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,skip_cleanup=True,key_size=18,randomize_doc_size=True,randomize_value=True,assert_crashes_on_load=True,doc_ops=update,maxttl=60,num_collections=20,durability=Majority,fragmentation=20,pc=2,crashes=10,sdk_client_pool=True -m rest'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Not seeing it on: Enterprise Edition 7.0.2 build 6683.
            Closing the defect.

            ritesh.agarwal Ritesh Agarwal added a comment - Not seeing it on: Enterprise Edition 7.0.2 build 6683. Closing the defect.

            Build couchbase-server-7.1.0-1096 contains ns_server commit 80e2872 with commit message:
            MB-46099 [cbcollect_info] Control the amount of outstanding IO.

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1096 contains ns_server commit 80e2872 with commit message: MB-46099 [cbcollect_info] Control the amount of outstanding IO.

            Build couchbase-server-7.0.1-5960 contains ns_server commit 80e2872 with commit message:
            MB-46099 [cbcollect_info] Control the amount of outstanding IO.

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-5960 contains ns_server commit 80e2872 with commit message: MB-46099 [cbcollect_info] Control the amount of outstanding IO.

            I've merged changes to mitigate this to chronicle. In addition, cbcollect_info (which is what triggered the issue in this test) should be less disruptive now. So I'm going to close the ticket.

            The investigation into a related MB-47169 is still ongoing.

            Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - I've merged changes to mitigate this to chronicle. In addition, cbcollect_info (which is what triggered the issue in this test) should be less disruptive now. So I'm going to close the ticket. The investigation into a related MB-47169 is still ongoing.

            Hello Aliaksey Artamonau, we do not run 130Node test quite often hence it is difficult to say on any recent occurrences.
            MB-47169 may be another instance of this issue...not too sure but the error seems to be similar.

            ritesh.agarwal Ritesh Agarwal added a comment - Hello Aliaksey Artamonau , we do not run 130Node test quite often hence it is difficult to say on any recent occurrences. MB-47169 may be another instance of this issue...not too sure but the error seems to be similar.

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty