Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50139

[10TB, Unbounded, KV]: buckets_shutdown_wait_failed on the node failed over gracefully and added back with full recovery followed by rebalanced.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 7.1.0
    • couchbase-bucket
    • 7.1.0-1913

    Description

      1. Create a 3 node cluster
      2. Create required buckets and collections.
      3. Create 40000000 items sequentially
      4. Update 40000000 RandonKey keys to create 50 percent fragmentation
      5. Create 40000000 items sequentially
      6. Update 40000000 RandonKey keys to create 50 percent fragmentation
      7. Rebalance in with Loading of docs
      8. Crash Magma/memc with Loading of docs
      9. Rebalance Out with Loading of docs
      10. Crash Magma/memc with Loading of docs
      11. Rebalance In_Out with Loading of docs
      12. Crash Magma/memc with Loading of docs
      13. Swap with Loading of docs
      14. Crash Magma/memc with Loading of docs
      15. Failover a node and RebalanceOut that node with loading in parallel
      16. Failover a node and FullRecovery that node
      17. During rebalance after failover + full recovery, rebalance failed with error: 

        Rebalance exited with reason {buckets_shutdown_wait_failed,
        		 
        [{'ns_1@172.23.110.66',
        		 
        {'EXIT',
        		 
        {old_buckets_shutdown_wait_failed,
        		 
        ["GleamBookUsers0"]}}}]}.
        		 
        Rebalance Operation Id = d5675b416af8bc4b53bfefc42faf8f24

      QE Test

      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-48225 Graceful Failover -> Full Recovery -> Rebalance, Rebalance failed with reason: buckets_shutdown_wait_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty