Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47169

[high-bucket] - 30 multi bucket test rebalance failed with buckets_cleanup_failed error

    XMLWordPrintable

Details

    Description

      Environment: 7.0.0-5295
      Test : 30 bucket test with all the components 
      Failed at : Rebalance step 
      Error message: 

      completionMessage":"Rebalance exited with reason
      Unknown macro:
      Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
      ."}

      Link to the job : http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/ 

      Steps of the test :

      1. Load the buckets with documents 
      2. Create n1ql indexes 
      3. Initialise XDCR (init_only_xdcr() )
      4. Creating the eventing functions
      5. Creating FTS indexes 
      6. Creating Analytics dataset
      7. Running rebalance for each phase as follows :
        1. KV rebalance 
          1. Rebalance in with mutations
          2. Rebalance swap 
          3. Rebalance out 
        2. Index rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        3. Eventing rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        4. CBAS rebalance 
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
      8. Backup
      9. FTS swap rebalance

      The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
      Cluster setup and the cluster details are mentioned in the screenshot attached below.

      Attachments

        1. Screenshot 2021-07-01 at 5.30.16 PM.png
          75 kB
          Jyotsna Nayak
        2. Screenshot 2021-07-01 at 5.31.24 PM.png
          33 kB
          Jyotsna Nayak
        3. slow_fsyncs_1s.png
          124 kB
          Aliaksey Artamonau
        4. slow_fsyncs_5s.png
          114 kB
          Aliaksey Artamonau
        5. loadavg.png
          148 kB
          Aliaksey Artamonau
        6. slow_fsyncs_1s_rate.png
          150 kB
          Aliaksey Artamonau
        7. screenshot-1.png
          42 kB
          Dave Finlay
        8. screenshot-2.png
          35 kB
          Dave Finlay
        9. screenshot-3.png
          49 kB
          Dave Finlay
        10. screenshot-4.png
          77 kB
          Dave Finlay
        11. screenshot-5.png
          39 kB
          Dave Finlay
        12. image-2021-08-19-18-50-23-794.png
          295 kB
          Jyotsna Nayak
        13. image-2021-08-19-19-36-41-983.png
          682 kB
          Jyotsna Nayak
        14. screenshot-6.png
          88 kB
          Dave Finlay
        15. screenshot-7.png
          67 kB
          Dave Finlay
        16. screenshot-8.png
          31 kB
          Dave Finlay
        17. screenshot-9.png
          29 kB
          Dave Finlay
        18. screenshot-10.png
          32 kB
          Dave Finlay
        19. Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          92 kB
          Aliaksey Artamonau
        20. screenshot-11.png
          47 kB
          Dave Finlay
        21. screenshot-12.png
          44 kB
          Dave Finlay
        22. screenshot-13.png
          38 kB
          Dave Finlay
        23. screenshot-14.png
          38 kB
          Dave Finlay
        24. screenshot-15.png
          45 kB
          Dave Finlay
        25. screenshot-16.png
          37 kB
          Dave Finlay
        26. screenshot-17.png
          37 kB
          Dave Finlay
        27. screenshot-18.png
          38 kB
          Dave Finlay
        28. screenshot-19.png
          33 kB
          Dave Finlay
        29. screenshot-20.png
          39 kB
          Dave Finlay

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-50856 Analytics ingestion rate dropped on build 7.1.0-2226

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-46099 130 node testing: Rebalance in 1 node in a 85 node cluster failed due to buckets_cleanup_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48194 [System Test] Rebalance exited with reason "bucket_cleanup_failed"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-47169
          # Subject Branch Project Status CR V
          169957,2 MB-47169: Make disk force bytes configurable neo cbas-core Status: MERGED +2 +1

          Activity

            People

              jyotsna.nayak Jyotsna Nayak (Inactive)
              jyotsna.nayak Jyotsna Nayak (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty