Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43576

Intermittent: Rebalance failed due to buckets_shutdown_wait_failed on failover followed by add back with full recovery.

    XMLWordPrintable

Details

    Description

      NOTE: We are hitting this issue in many other rebalance functional tests.

      Steps:

      1. Create a 15 node cluster
      2. Create required buckets and collections.
      3. Create 1000000 items sequentially
      4. Create 1000000 random keys
      5. Update 2000000 random keys to create 50 percent fragmentation
      6. Rebalance in with Loading of docs
      7. Rebalance Out with Loading of docs
      8. Rebalance In_Out with Loading of docs
      9. Swap with Loading of docs
      10. Failover a node and RebalanceOut that node with loading in parallel
      11. Failover a node and FullRecovery that node

      Rebalance failed at last step mentioned above.

      Rebalance exited with reason {buckets_shutdown_wait_failed,
      		 
      [{'ns_1@172.23.121.140',
      		 
      {'EXIT',
      		 
      {old_buckets_shutdown_wait_failed,
      		 
      ["GleamBook"]}}}]}.
      		 
      Rebalance Operation Id = ff702a3d90b5cee331aa500b825c52e9
      		 
       
      		 
       
      		 
      Failed to wait deletion of some buckets on some nodes: [{'ns_1@172.23.121.140',
      		 
      {'EXIT',
      		 
      {old_buckets_shutdown_wait_failed,
      		 
      ["GleamBook"]}}}]

      Multiple rebalance reties also failed.

      On build 4122 this step did passed in the volume test.

      Attachments

        1. 172.23.121.48.txt
          132 kB
        2. 172.23.121.141.txt
          132 kB
        3. 172.23.121.140.txt
          132 kB
        4. 172.23.121.139.txt
          161 kB
        5. 172.23.121.136.txt
          37 kB
        6. 172.23.121.135.txt
          161 kB
        7. 172.23.121.134.txt
          164 kB
        8. 172.23.121.133.txt
          161 kB
        9. 172.23.121.132.txt
          132 kB
        10. 172.23.121.131.txt
          162 kB
        11. 172.23.121.130.txt
          161 kB
        12. 172.23.121.129.txt
          145 kB
        13. 172.23.121.128.txt
          37 kB
        14. 172.23.121.127.txt
          37 kB
        15. 172.23.121.126.txt
          166 kB
        16. 172.23.121.124.txt
          161 kB
        17. 172.23.121.123.txt
          161 kB
        18. 172.23.121.116.txt
          167 kB
        19. 172.23.121.115.txt
          37 kB
        20. 172.23.120.170.txt
          162 kB

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-43373 Exceptions thrown in FollyExecutorPool tasks can hang KV-Engine tasks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-43576
          # Subject Branch Project Status CR V
          143254,2 MB-43576: Improve magma deinitializing logging master kv_engine Status: MERGED +2 +1
          143275,4 MB-43576 magma: Add log message for closing db master magma Status: MERGED +2 +1

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty