Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
7.0.0
-
Untriaged
-
-
1
-
No
Description
Environment: 7.0.0-5295
Test : 30 bucket test with all the components
Failed at : Rebalance step
Error message:
completionMessage":"Rebalance exited with reason
Unknown macro:
Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']}
."}
Link to the job : http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/
Steps of the test :
- Load the buckets with documents
- Create n1ql indexes
- Initialise XDCR (init_only_xdcr() )
- Creating the eventing functions
- Creating FTS indexes
- Creating Analytics dataset
- Running rebalance for each phase as follows :
- KV rebalance
- Rebalance in with mutations
- Rebalance swap
- Rebalance out
- Index rebalance
- Rebalance in
- Rebalance swap
- Rebalance Out
- Eventing rebalance
- Rebalance in
- Rebalance swap
- Rebalance Out
- CBAS rebalance
- Rebalance in
- Rebalance swap
- Rebalance Out
- KV rebalance
- Backup
- FTS swap rebalance
The test failed when Eventing Swap rebalance was being executed. (Marked in red)
Cluster setup and the cluster details are mentioned in the screenshot attached below.
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Link to Log File, atop/blg, CBCollectInfo, Core dump | https://issues.couchbase.com/browse/MB-46099 |
| Affects Version/s | 7.0.0 [ 17233 ] |
| Summary | 30 multi bucket test rebalance failed with cleanup issues | 30 multi bucket test rebalance failed with buckets_cleanup_failed |
| Summary | 30 multi bucket test rebalance failed with buckets_cleanup_failed | 30 multi bucket test rebalance failed with buckets_cleanup_failed error |
| Summary | 30 multi bucket test rebalance failed with buckets_cleanup_failed error | [high-bucket] - 30 multi bucket test rebalance failed with buckets_cleanup_failed error |
| Component/s | ns_server [ 10019 ] |
| Labels | high-bucket-density |
| Assignee | Meni Hillel [ JIRAUSER25407 ] |
| Assignee | Meni Hillel [ JIRAUSER25407 ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Labels | high-bucket-density | high-bucket-density performance |
| Description |
Environment: 7.0.0-5295
Test : 30 bucket test with all the components Failed at : Rebalance step Error message: {quote}completionMessage":"Rebalance exited with reason {buckets_cleanup_failed,['ns_1@172.23.96.20']}."} {quote} Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/] |
Environment: 7.0.0-5295
Test : 30 bucket test with all the components Failed at : Rebalance step Error message: {quote}completionMessage":"Rebalance exited with reason Unknown macro: \{buckets_cleanup_failed,['ns_1@172.23.96.20']} ."} {quote} Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/] h3. Steps of the test : # Load the buckets with documents # Create n1ql indexes # Initialise XDCR (init_only_xdcr() ) # Creating the eventing functions # Creating FTS indexes # Creating Analytics dataset # Running rebalance for each phase as follows : # KV rebalance # Rebalance in with mutations # Rebalance swap # Rebalance out # Index rebalance # Rebalance in # Rebalance swap # Rebalance Out # Eventing rebalance # Rebalance in # Rebalance swap # Rebalance Out # CBAS rebalance # Rebalance in # Rebalance swap # Rebalance Out # Backup # FTS swap rebalance The test failed when Eventing Swap rebalance was being executed. |
| Description |
Environment: 7.0.0-5295
Test : 30 bucket test with all the components Failed at : Rebalance step Error message: {quote}completionMessage":"Rebalance exited with reason Unknown macro: \{buckets_cleanup_failed,['ns_1@172.23.96.20']} ."} {quote} Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/] h3. Steps of the test : # Load the buckets with documents # Create n1ql indexes # Initialise XDCR (init_only_xdcr() ) # Creating the eventing functions # Creating FTS indexes # Creating Analytics dataset # Running rebalance for each phase as follows : # KV rebalance # Rebalance in with mutations # Rebalance swap # Rebalance out # Index rebalance # Rebalance in # Rebalance swap # Rebalance Out # Eventing rebalance # Rebalance in # Rebalance swap # Rebalance Out # CBAS rebalance # Rebalance in # Rebalance swap # Rebalance Out # Backup # FTS swap rebalance The test failed when Eventing Swap rebalance was being executed. |
Environment: 7.0.0-5295
Test : 30 bucket test with all the components Failed at : Rebalance step Error message: {quote}completionMessage":"Rebalance exited with reason Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']} ."} {quote} Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/] h3. Steps of the test : # Load the buckets with documents # Create n1ql indexes # Initialise XDCR (init_only_xdcr() ) # Creating the eventing functions # Creating FTS indexes # Creating Analytics dataset # Running rebalance for each phase as follows : ## KV rebalance ### Rebalance in with mutations ### Rebalance swap ### Rebalance out ## Index rebalance ### Rebalance in ### Rebalance swap ### Rebalance Out ## Eventing rebalance ### Rebalance in ### {color:#de350b}Rebalance swap{color} ### Rebalance Out ## CBAS rebalance ### Rebalance in ### Rebalance swap ### Rebalance Out # Backup # FTS swap rebalance The test failed when Eventing Swap rebalance was being executed. (Marked in red) |
| Attachment | Screenshot 2021-07-01 at 5.30.16 PM.png [ 148473 ] |
| Attachment | Screenshot 2021-07-01 at 5.31.24 PM.png [ 148474 ] |
| Description |
Environment: 7.0.0-5295
Test : 30 bucket test with all the components Failed at : Rebalance step Error message: {quote}completionMessage":"Rebalance exited with reason Unknown macro: {buckets_cleanup_failed,['ns_1@172.23.96.20']} ."} {quote} Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/] h3. Steps of the test : # Load the buckets with documents # Create n1ql indexes # Initialise XDCR (init_only_xdcr() ) # Creating the eventing functions # Creating FTS indexes # Creating Analytics dataset # Running rebalance for each phase as follows : ## KV rebalance ### Rebalance in with mutations ### Rebalance swap ### Rebalance out ## Index rebalance ### Rebalance in ### Rebalance swap ### Rebalance Out ## Eventing rebalance ### Rebalance in ### {color:#de350b}Rebalance swap{color} ### Rebalance Out ## CBAS rebalance ### Rebalance in ### Rebalance swap ### Rebalance Out # Backup # FTS swap rebalance The test failed when Eventing Swap rebalance was being executed. (Marked in red) |
Environment: 7.0.0-5295
Test : 30 bucket test with all the components Failed at : Rebalance step Error message: {quote}completionMessage":"Rebalance exited with reason Unknown macro: Unknown macro: \{buckets_cleanup_failed,['ns_1@172.23.96.20']} ."} {quote} Link to the job : [http://perf.jenkins.couchbase.com/view/Eventing/job/themis_multibucket/102/] h3. Steps of the test : # Load the buckets with documents # Create n1ql indexes # Initialise XDCR (init_only_xdcr() ) # Creating the eventing functions # Creating FTS indexes # Creating Analytics dataset # Running rebalance for each phase as follows : ## KV rebalance ### Rebalance in with mutations ### Rebalance swap ### Rebalance out ## Index rebalance ### Rebalance in ### Rebalance swap ### Rebalance Out ## Eventing rebalance ### Rebalance in ### {color:#de350b}Rebalance swap{color} ### Rebalance Out ## CBAS rebalance ### Rebalance in ### Rebalance swap ### Rebalance Out # Backup # FTS swap rebalance The test failed when Eventing Swap rebalance was being executed. (Marked in red) Cluster setup and the cluster details are mentioned in the screenshot attached below. |
| Assignee | Jyotsna Nayak [ jyotsna.nayak ] | Meni Hillel [ JIRAUSER25407 ] |
| Assignee | Meni Hillel [ JIRAUSER25407 ] | Artem Stemkovski [ artem ] |
| Fix Version/s | 7.0.1 [ 17104 ] |
| Assignee | Artem Stemkovski [ artem ] | Aliaksey Artamonau [ aliaksey artamonau ] |
| Assignee | Aliaksey Artamonau [ aliaksey artamonau ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Assignee | Jyotsna Nayak [ jyotsna.nayak ] | Aliaksey Artamonau [ aliaksey artamonau ] |
| Attachment | slow_fsyncs_1s.png [ 151526 ] |
| Attachment | slow_fsyncs_5s.png [ 151527 ] |
| Attachment | loadavg.png [ 151528 ] |
| Attachment | slow_fsyncs_1s_rate.png [ 151529 ] |
| Component/s | analytics [ 12717 ] | |
| Component/s | fts [ 12726 ] |
| Assignee | Aliaksey Artamonau [ aliaksey artamonau ] | Meni Hillel [ JIRAUSER25407 ] |
| Assignee | Meni Hillel [ JIRAUSER25407 ] | Dave Finlay [ dfinlay ] |
| Fix Version/s | 7.0.2 [ 18012 ] | |
| Fix Version/s | 7.0.1 [ 17104 ] |
| Fix Version/s | Neo [ 17615 ] | |
| Fix Version/s | 7.0.2 [ 18012 ] |
| Rank | Ranked higher |
| Attachment | screenshot-1.png [ 154426 ] |
| Attachment | screenshot-2.png [ 154427 ] |
| Attachment | screenshot-3.png [ 154428 ] |
| Attachment | screenshot-4.png [ 154429 ] |
| Attachment | screenshot-5.png [ 154432 ] |
| Assignee | Dave Finlay [ dfinlay ] | Till Westmann [ till ] |
| Assignee | Till Westmann [ till ] | Murtadha Hubail [ murtadha.hubail ] |
| Assignee | Murtadha Hubail [ murtadha.hubail ] | Dave Finlay [ dfinlay ] |
| Assignee | Dave Finlay [ dfinlay ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Attachment | Screenshot 2021-08-19 at 6.49.31 PM.png [ 155616 ] |
| Attachment | Screenshot 2021-08-19 at 6.49.31 PM.png [ 155616 ] |
| Attachment | image-2021-08-19-18-50-23-794.png [ 155617 ] |
| Attachment | image-2021-08-19-19-36-41-983.png [ 155629 ] |
| Labels | high-bucket-density performance | high-bucket-density performance triaged |
| Assignee | Jyotsna Nayak [ jyotsna.nayak ] | Dave Finlay [ dfinlay ] |
| Attachment | screenshot-6.png [ 158904 ] |
| Attachment | screenshot-7.png [ 158909 ] |
| Attachment | screenshot-8.png [ 158910 ] |
| Attachment | screenshot-9.png [ 158911 ] |
| Attachment | screenshot-10.png [ 158912 ] |
| Assignee | Dave Finlay [ dfinlay ] | Murtadha Hubail [ murtadha.hubail ] |
| Assignee | Murtadha Hubail [ murtadha.hubail ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Rank | Ranked higher |
| Assignee | Jyotsna Nayak [ jyotsna.nayak ] | Dave Finlay [ dfinlay ] |
| Attachment | Screenshot 2021-12-01 at 21-22-09 Chronicle Node - Grafana.png [ 171522 ] |
| Rank | Ranked higher |
| Attachment | screenshot-11.png [ 175070 ] |
| Attachment | screenshot-12.png [ 175071 ] |
| Attachment | screenshot-13.png [ 175072 ] |
| Attachment | screenshot-14.png [ 175073 ] |
| Priority | Major [ 3 ] | Critical [ 2 ] |
| Assignee | Dave Finlay [ dfinlay ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Rank | Ranked higher |
| Assignee | Jyotsna Nayak [ jyotsna.nayak ] | Dave Finlay [ dfinlay ] |
| Is this a Regression? | Unknown [ 10452 ] | No [ 10451 ] |
| Attachment | screenshot-15.png [ 177369 ] |
| Attachment | screenshot-16.png [ 177370 ] |
| Attachment | screenshot-17.png [ 177371 ] |
| Attachment | screenshot-18.png [ 177372 ] |
| Attachment | screenshot-19.png [ 177373 ] |
| Attachment | screenshot-20.png [ 177374 ] |
| Assignee | Dave Finlay [ dfinlay ] | Till Westmann [ till ] |
| Component/s | fts [ 12726 ] |
| Component/s | ns_server [ 10019 ] |
| Assignee | Till Westmann [ till ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Assignee | Jyotsna Nayak [ jyotsna.nayak ] | Murtadha Hubail [ murtadha.hubail ] |
| Assignee | Murtadha Hubail [ murtadha.hubail ] | Jyotsna Nayak [ jyotsna.nayak ] |
| Status | Resolved [ 5 ] | Closed [ 6 ] |