Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47169

[high-bucket] - 30 multi bucket test rebalance failed with buckets_cleanup_failed error

    XMLWordPrintable

Details

    Description

      Environment: 7.0.0-5295
      Test : 30 bucket test with all the components 
      Failed at : Rebalance step 
      Error message: 

      completionMessage":"Rebalance exited with reason
      Unknown macro:
      Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
      ."}

      Link to the job : http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/ 

      Steps of the test :

      1. Load the buckets with documents 
      2. Create n1ql indexes 
      3. Initialise XDCR (init_only_xdcr() )
      4. Creating the eventing functions
      5. Creating FTS indexes 
      6. Creating Analytics dataset
      7. Running rebalance for each phase as follows :
        1. KV rebalance 
          1. Rebalance in with mutations
          2. Rebalance swap 
          3. Rebalance out 
        2. Index rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        3. Eventing rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        4. CBAS rebalance 
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
      8. Backup
      9. FTS swap rebalance

      The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
      Cluster setup and the cluster details are mentioned in the screenshot attached below.

      Attachments

        1. image-2021-08-19-18-50-23-794.png
          image-2021-08-19-18-50-23-794.png
          295 kB
        2. image-2021-08-19-19-36-41-983.png
          image-2021-08-19-19-36-41-983.png
          682 kB
        3. loadavg.png
          loadavg.png
          148 kB
        4. screenshot-1.png
          screenshot-1.png
          42 kB
        5. screenshot-10.png
          screenshot-10.png
          32 kB
        6. screenshot-11.png
          screenshot-11.png
          47 kB
        7. screenshot-12.png
          screenshot-12.png
          44 kB
        8. screenshot-13.png
          screenshot-13.png
          38 kB
        9. screenshot-14.png
          screenshot-14.png
          38 kB
        10. screenshot-15.png
          screenshot-15.png
          45 kB
        11. screenshot-16.png
          screenshot-16.png
          37 kB
        12. screenshot-17.png
          screenshot-17.png
          37 kB
        13. screenshot-18.png
          screenshot-18.png
          38 kB
        14. screenshot-19.png
          screenshot-19.png
          33 kB
        15. screenshot-2.png
          screenshot-2.png
          35 kB
        16. screenshot-20.png
          screenshot-20.png
          39 kB
        17. Screenshot 2021-07-01 at 5.30.16 PM.png
          Screenshot 2021-07-01 at 5.30.16 PM.png
          75 kB
        18. Screenshot 2021-07-01 at 5.31.24 PM.png
          Screenshot 2021-07-01 at 5.31.24 PM.png
          33 kB
        19. Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          92 kB
        20. screenshot-3.png
          screenshot-3.png
          49 kB
        21. screenshot-4.png
          screenshot-4.png
          77 kB
        22. screenshot-5.png
          screenshot-5.png
          39 kB
        23. screenshot-6.png
          screenshot-6.png
          88 kB
        24. screenshot-7.png
          screenshot-7.png
          67 kB
        25. screenshot-8.png
          screenshot-8.png
          31 kB
        26. screenshot-9.png
          screenshot-9.png
          29 kB
        27. slow_fsyncs_1s_rate.png
          slow_fsyncs_1s_rate.png
          150 kB
        28. slow_fsyncs_1s.png
          slow_fsyncs_1s.png
          124 kB
        29. slow_fsyncs_5s.png
          slow_fsyncs_5s.png
          114 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-50856 Analytics ingestion rate dropped on build 7.1.0-2226

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-46099 130 node testing: Rebalance in 1 node in a 85 node cluster failed due to buckets_cleanup_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48194 [System Test] Rebalance exited with reason "bucket_cleanup_failed"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-47169
          # Subject Branch Project Status CR V
          169957,2 MB-47169: Make disk force bytes configurable neo cbas-core Status: MERGED +2 +1

          Activity

            People

              jyotsna.nayak Jyotsna Nayak
              jyotsna.nayak Jyotsna Nayak
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty