Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47169

[high-bucket] - 30 multi bucket test rebalance failed with buckets_cleanup_failed error

    XMLWordPrintable

Details

    Description

      Environment: 7.0.0-5295
      Test : 30 bucket test with all the components 
      Failed at : Rebalance step 
      Error message: 

      completionMessage":"Rebalance exited with reason
      Unknown macro:
      Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
      ."}

      Link to the job : http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/ 

      Steps of the test :

      1. Load the buckets with documents 
      2. Create n1ql indexes 
      3. Initialise XDCR (init_only_xdcr() )
      4. Creating the eventing functions
      5. Creating FTS indexes 
      6. Creating Analytics dataset
      7. Running rebalance for each phase as follows :
        1. KV rebalance 
          1. Rebalance in with mutations
          2. Rebalance swap 
          3. Rebalance out 
        2. Index rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        3. Eventing rebalance
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
        4. CBAS rebalance 
          1. Rebalance in 
          2. Rebalance swap
          3. Rebalance Out 
      8. Backup
      9. FTS swap rebalance

      The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
      Cluster setup and the cluster details are mentioned in the screenshot attached below.

      Attachments

        1. image-2021-08-19-18-50-23-794.png
          image-2021-08-19-18-50-23-794.png
          295 kB
        2. image-2021-08-19-19-36-41-983.png
          image-2021-08-19-19-36-41-983.png
          682 kB
        3. loadavg.png
          loadavg.png
          148 kB
        4. screenshot-1.png
          screenshot-1.png
          42 kB
        5. screenshot-10.png
          screenshot-10.png
          32 kB
        6. screenshot-11.png
          screenshot-11.png
          47 kB
        7. screenshot-12.png
          screenshot-12.png
          44 kB
        8. screenshot-13.png
          screenshot-13.png
          38 kB
        9. screenshot-14.png
          screenshot-14.png
          38 kB
        10. screenshot-15.png
          screenshot-15.png
          45 kB
        11. screenshot-16.png
          screenshot-16.png
          37 kB
        12. screenshot-17.png
          screenshot-17.png
          37 kB
        13. screenshot-18.png
          screenshot-18.png
          38 kB
        14. screenshot-19.png
          screenshot-19.png
          33 kB
        15. screenshot-2.png
          screenshot-2.png
          35 kB
        16. screenshot-20.png
          screenshot-20.png
          39 kB
        17. Screenshot 2021-07-01 at 5.30.16 PM.png
          Screenshot 2021-07-01 at 5.30.16 PM.png
          75 kB
        18. Screenshot 2021-07-01 at 5.31.24 PM.png
          Screenshot 2021-07-01 at 5.31.24 PM.png
          33 kB
        19. Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png
          92 kB
        20. screenshot-3.png
          screenshot-3.png
          49 kB
        21. screenshot-4.png
          screenshot-4.png
          77 kB
        22. screenshot-5.png
          screenshot-5.png
          39 kB
        23. screenshot-6.png
          screenshot-6.png
          88 kB
        24. screenshot-7.png
          screenshot-7.png
          67 kB
        25. screenshot-8.png
          screenshot-8.png
          31 kB
        26. screenshot-9.png
          screenshot-9.png
          29 kB
        27. slow_fsyncs_1s_rate.png
          slow_fsyncs_1s_rate.png
          150 kB
        28. slow_fsyncs_1s.png
          slow_fsyncs_1s.png
          124 kB
        29. slow_fsyncs_5s.png
          slow_fsyncs_5s.png
          114 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jyotsna.nayak Jyotsna Nayak created issue -
            wayne Wayne Siu made changes -
            Field Original Value New Value
            Link to Log File, atop/blg, CBCollectInfo, Core dump https://issues.couchbase.com/browse/MB-46099
            wayne Wayne Siu made changes -
            Link This issue relates to MB-46099 [ MB-46099 ]
            wayne Wayne Siu made changes -
            Affects Version/s 7.0.0 [ 17233 ]
            wayne Wayne Siu made changes -
            Summary 30 multi bucket test rebalance failed with cleanup issues 30 multi bucket test rebalance failed with buckets_cleanup_failed
            wayne Wayne Siu made changes -
            Summary 30 multi bucket test rebalance failed with buckets_cleanup_failed 30 multi bucket test rebalance failed with buckets_cleanup_failed error
            wayne Wayne Siu made changes -
            Summary 30 multi bucket test rebalance failed with buckets_cleanup_failed error [high-bucket] - 30 multi bucket test rebalance failed with buckets_cleanup_failed error
            wayne Wayne Siu made changes -
            Component/s ns_server [ 10019 ]
            wayne Wayne Siu made changes -
            Labels high-bucket-density
            wayne Wayne Siu made changes -
            Assignee Meni Hillel [ JIRAUSER25407 ]
            wayne Wayne Siu made changes -
            Assignee Meni Hillel [ JIRAUSER25407 ] Jyotsna Nayak [ jyotsna.nayak ]
            wayne Wayne Siu made changes -
            Labels high-bucket-density high-bucket-density performance
            jyotsna.nayak Jyotsna Nayak made changes -
            Description Environment: 7.0.0-5295
             Test : 30 bucket test with all the components 
             Failed at : Rebalance step 
             Error message: 
            {quote}completionMessage":"Rebalance exited with reason {buckets_cleanup_failed,['ns_1@172.23.96.20']}."}


            {quote}
            Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/
            Environment: 7.0.0-5295
             Test : 30 bucket test with all the components 
             Failed at : Rebalance step 
             Error message: 
            {quote}completionMessage":"Rebalance exited with reason
            Unknown macro: \{buckets_cleanup_failed,['ns_1@172.23.96.20']}
            ."}
            {quote}
            Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/


            h3. Steps of the test :
             # Load the buckets with documents 
             # Create n1ql indexes 
             # Initialise XDCR (init_only_xdcr() )
             # Creating the eventing functions
             # Creating FTS indexes 
             # Creating Analytics dataset
             # Running rebalance for each phase as follows :
             # KV rebalance 
             # Rebalance in with mutations
             # Rebalance swap 
             # Rebalance out 


             # Index rebalance
             # Rebalance in 
             # Rebalance swap
             # Rebalance Out 


             # Eventing rebalance
             # Rebalance in 
             # Rebalance swap
             # Rebalance Out 


             # CBAS rebalance 
             # Rebalance in 
             # Rebalance swap
             # Rebalance Out 


             # Backup
             # FTS swap rebalance

            The test failed when Eventing Swap rebalance was being executed. 
            jyotsna.nayak Jyotsna Nayak made changes -
            Description Environment: 7.0.0-5295
             Test : 30 bucket test with all the components 
             Failed at : Rebalance step 
             Error message: 
            {quote}completionMessage":"Rebalance exited with reason
            Unknown macro: \{buckets_cleanup_failed,['ns_1@172.23.96.20']}
            ."}
            {quote}
            Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/


            h3. Steps of the test :
             # Load the buckets with documents 
             # Create n1ql indexes 
             # Initialise XDCR (init_only_xdcr() )
             # Creating the eventing functions
             # Creating FTS indexes 
             # Creating Analytics dataset
             # Running rebalance for each phase as follows :
             # KV rebalance 
             # Rebalance in with mutations
             # Rebalance swap 
             # Rebalance out 


             # Index rebalance
             # Rebalance in 
             # Rebalance swap
             # Rebalance Out 


             # Eventing rebalance
             # Rebalance in 
             # Rebalance swap
             # Rebalance Out 


             # CBAS rebalance 
             # Rebalance in 
             # Rebalance swap
             # Rebalance Out 


             # Backup
             # FTS swap rebalance

            The test failed when Eventing Swap rebalance was being executed. 
            Environment: 7.0.0-5295
             Test : 30 bucket test with all the components 
             Failed at : Rebalance step 
             Error message: 
            {quote}completionMessage":"Rebalance exited with reason
             Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
             ."}
            {quote}
            Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/
            h3. Steps of the test :
             # Load the buckets with documents 
             # Create n1ql indexes 
             # Initialise XDCR (init_only_xdcr() )
             # Creating the eventing functions
             # Creating FTS indexes 
             # Creating Analytics dataset
             # Running rebalance for each phase as follows :
             ## KV rebalance 
             ### Rebalance in with mutations
             ### Rebalance swap 
             ### Rebalance out 
             ## Index rebalance
             ### Rebalance in 
             ### Rebalance swap
             ### Rebalance Out 
             ## Eventing rebalance
             ### Rebalance in 
             ### {color:#de350b}Rebalance swap{color}
             ### Rebalance Out 
             ## CBAS rebalance 
             ### Rebalance in 
             ### Rebalance swap
             ### Rebalance Out 
             # Backup
             # FTS swap rebalance

            The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
            jyotsna.nayak Jyotsna Nayak made changes -
            jyotsna.nayak Jyotsna Nayak made changes -
            jyotsna.nayak Jyotsna Nayak made changes -
            Description Environment: 7.0.0-5295
             Test : 30 bucket test with all the components 
             Failed at : Rebalance step 
             Error message: 
            {quote}completionMessage":"Rebalance exited with reason
             Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
             ."}
            {quote}
            Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/
            h3. Steps of the test :
             # Load the buckets with documents 
             # Create n1ql indexes 
             # Initialise XDCR (init_only_xdcr() )
             # Creating the eventing functions
             # Creating FTS indexes 
             # Creating Analytics dataset
             # Running rebalance for each phase as follows :
             ## KV rebalance 
             ### Rebalance in with mutations
             ### Rebalance swap 
             ### Rebalance out 
             ## Index rebalance
             ### Rebalance in 
             ### Rebalance swap
             ### Rebalance Out 
             ## Eventing rebalance
             ### Rebalance in 
             ### {color:#de350b}Rebalance swap{color}
             ### Rebalance Out 
             ## CBAS rebalance 
             ### Rebalance in 
             ### Rebalance swap
             ### Rebalance Out 
             # Backup
             # FTS swap rebalance

            The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
            Environment: 7.0.0-5295
             Test : 30 bucket test with all the components 
             Failed at : Rebalance step 
             Error message: 
            {quote}completionMessage":"Rebalance exited with reason
             Unknown macro:
            Unknown macro: \{buckets_cleanup_failed,['ns_1@172.23.96.20']}
            ."}
            {quote}
            Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/
            h3. Steps of the test :
             # Load the buckets with documents 
             # Create n1ql indexes 
             # Initialise XDCR (init_only_xdcr() )
             # Creating the eventing functions
             # Creating FTS indexes 
             # Creating Analytics dataset
             # Running rebalance for each phase as follows :
             ## KV rebalance 
             ### Rebalance in with mutations
             ### Rebalance swap 
             ### Rebalance out 
             ## Index rebalance
             ### Rebalance in 
             ### Rebalance swap
             ### Rebalance Out 
             ## Eventing rebalance
             ### Rebalance in 
             ### {color:#de350b}Rebalance swap{color}
             ### Rebalance Out 
             ## CBAS rebalance 
             ### Rebalance in 
             ### Rebalance swap
             ### Rebalance Out 
             # Backup
             # FTS swap rebalance

            The test failed when Eventing Swap rebalance was being executed. (Marked in red) 
            Cluster setup and the cluster details are mentioned in the screenshot attached below.
            wayne Wayne Siu made changes -
            Assignee Jyotsna Nayak [ jyotsna.nayak ] Meni Hillel [ JIRAUSER25407 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Assignee Meni Hillel [ JIRAUSER25407 ] Artem Stemkovski [ artem ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            artem Artem Stemkovski made changes -
            Assignee Artem Stemkovski [ artem ] Aliaksey Artamonau [ aliaksey artamonau ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Assignee Aliaksey Artamonau [ aliaksey artamonau ] Jyotsna Nayak [ jyotsna.nayak ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Assignee Jyotsna Nayak [ jyotsna.nayak ] Aliaksey Artamonau [ aliaksey artamonau ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Attachment slow_fsyncs_1s.png [ 151526 ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Attachment slow_fsyncs_5s.png [ 151527 ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Attachment loadavg.png [ 151528 ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Attachment slow_fsyncs_1s_rate.png [ 151529 ]
            Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
            Component/s analytics [ 12717 ]
            Component/s fts [ 12726 ]
            dfinlay Dave Finlay made changes -
            Assignee Aliaksey Artamonau [ aliaksey artamonau ] Meni Hillel [ JIRAUSER25407 ]
            dfinlay Dave Finlay made changes -
            Assignee Meni Hillel [ JIRAUSER25407 ] Dave Finlay [ dfinlay ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            Fix Version/s 7.0.1 [ 17104 ]
            abhinav Abhinav Dangeti made changes -
            Fix Version/s Neo [ 17615 ]
            Fix Version/s 7.0.2 [ 18012 ]
            till Till Westmann made changes -
            Rank Ranked higher
            dfinlay Dave Finlay made changes -
            Attachment screenshot-1.png [ 154426 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-2.png [ 154427 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-3.png [ 154428 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-4.png [ 154429 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-5.png [ 154432 ]
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Till Westmann [ till ]
            till Till Westmann made changes -
            Assignee Till Westmann [ till ] Murtadha Hubail [ murtadha.hubail ]
            murtadha.hubail Murtadha Hubail made changes -
            Assignee Murtadha Hubail [ murtadha.hubail ] Dave Finlay [ dfinlay ]
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Jyotsna Nayak [ jyotsna.nayak ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Attachment Screenshot 2021-08-19 at 6.49.31 PM.png [ 155616 ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Attachment Screenshot 2021-08-19 at 6.49.31 PM.png [ 155616 ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Attachment image-2021-08-19-18-50-23-794.png [ 155617 ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Attachment image-2021-08-19-19-36-41-983.png [ 155629 ]
            till Till Westmann made changes -
            Labels high-bucket-density performance high-bucket-density performance triaged
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Link This issue relates to MB-48194 [ MB-48194 ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Assignee Jyotsna Nayak [ jyotsna.nayak ] Dave Finlay [ dfinlay ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-6.png [ 158904 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-7.png [ 158909 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-8.png [ 158910 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-9.png [ 158911 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-10.png [ 158912 ]
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Murtadha Hubail [ murtadha.hubail ]
            murtadha.hubail Murtadha Hubail made changes -
            Assignee Murtadha Hubail [ murtadha.hubail ] Jyotsna Nayak [ jyotsna.nayak ]
            till Till Westmann made changes -
            Rank Ranked higher
            jyotsna.nayak Jyotsna Nayak made changes -
            Assignee Jyotsna Nayak [ jyotsna.nayak ] Dave Finlay [ dfinlay ]
            till Till Westmann made changes -
            Rank Ranked higher
            dfinlay Dave Finlay made changes -
            Attachment screenshot-11.png [ 175070 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-12.png [ 175071 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-13.png [ 175072 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-14.png [ 175073 ]
            dfinlay Dave Finlay made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Jyotsna Nayak [ jyotsna.nayak ]
            till Till Westmann made changes -
            Link This issue relates to MB-50856 [ MB-50856 ]
            till Till Westmann made changes -
            Rank Ranked higher
            jyotsna.nayak Jyotsna Nayak made changes -
            Assignee Jyotsna Nayak [ jyotsna.nayak ] Dave Finlay [ dfinlay ]
            mary.roth Mary Roth made changes -
            Is this a Regression? Unknown [ 10452 ] No [ 10451 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-15.png [ 177369 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-16.png [ 177370 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-17.png [ 177371 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-18.png [ 177372 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-19.png [ 177373 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-20.png [ 177374 ]
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Till Westmann [ till ]
            dfinlay Dave Finlay made changes -
            Component/s fts [ 12726 ]
            dfinlay Dave Finlay made changes -
            Component/s ns_server [ 10019 ]
            murtadha.hubail Murtadha Hubail made changes -
            Assignee Till Westmann [ till ] Jyotsna Nayak [ jyotsna.nayak ]
            murtadha.hubail Murtadha Hubail made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            jyotsna.nayak Jyotsna Nayak made changes -
            Assignee Jyotsna Nayak [ jyotsna.nayak ] Murtadha Hubail [ murtadha.hubail ]
            murtadha.hubail Murtadha Hubail made changes -
            Assignee Murtadha Hubail [ murtadha.hubail ] Jyotsna Nayak [ jyotsna.nayak ]
            wayne Wayne Siu made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              jyotsna.nayak Jyotsna Nayak
              jyotsna.nayak Jyotsna Nayak
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty