Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40421

Rebalance analytics nodes fails with HYR0080: Cannot create index because it already exists

    XMLWordPrintable

    Details

      Description

      Rebalance of analytics node fails with :
      Add and remove analytics node and then rebalance:

      [2020-07-12T14:37:30-07:00, sequoiatools/couchbase-cli:6.5:b52b88] server-add -c 172.23.108.103:8091 --server-add https://172.23.99.20 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics
      [2020-07-12T14:38:12-07:00, sequoiatools/couchbase-cli:6.5:afaf7f] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.239 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.239 -u Administrator -p password]
       
      docker logs afaf7f
      docker start afaf7f
       
      sWARNING: couchbase-cli version 6.5.0-3216-enterprise does not match couchbase server version 6.6.0-7873-enterprise
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      [{'ns_1@172.23.97.122',
           {exit,
               {{nodedown,'ns_1@172.23.97.122'},
                {gen_server,call,
                    [{ns_config_rep,'ns_1@172.23.97.122'},
                     synchronize_everything,infinity]}}}}]
      [ns_server:error,2020-07-10T10:33:06.660-07:00,ns_1@172.23.108.103:service_rebalancer-cbas<0.9798.462>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.9793.462>,
                                     {rebalance_failed,
                                      {service_error,
                                       <<"Rebalance d8e520f7fd4f00e1c531f8df5540817e failed: HYR0080: Cannot create index because it already exists">>}}}
      [user:error,2020-07-10T10:33:06.663-07:00,ns_1@172.23.108.103:<0.24054.249>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,cbas,
                                    {worker_died,
                                     {'EXIT',<0.9793.462>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"Rebalance d8e520f7fd4f00e1c531f8df5540817e failed: HYR0080: Cannot create index because it already exists">>}}}}}.
      Rebalance Operation Id = b23e6cee01bee5b04faf7cc2bcf5053c
      [ns_server:error,2020-07-10T11:01:36.380-07:00,ns_1@172.23.108.103:service_rebalancer-cbas<0.11839.480>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.696.480>,
                                     {rebalance_failed,
                                      {service_error,
                                       <<"Rebalance 4e723c73ef045aa36646691a7e6045d9 failed: HYR0080: Cannot create index because it already exists">>}}}
      
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            ritam.sharma Ritam Sharma created issue -
            ritam.sharma Ritam Sharma made changes -
            Field Original Value New Value
            Link to Log File, atop/blg, CBCollectInfo, Core dump url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.137.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.155.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.157.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.5.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.61.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.67.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.69.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.70.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.87.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.104.88.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.105.107.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.106.100.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.106.188.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.108.103.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.120.245.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.96.148.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.96.251.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.96.252.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.96.253.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.96.95.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.97.119.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.97.121.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.97.122.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.97.239.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.97.242.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.98.135.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.99.11.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.99.20.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.99.21.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1594595088/collectinfo-2020-07-12T230451-ns_1%40172.23.99.25.zip
            till Till Westmann made changes -
            Assignee Till Westmann [ till ] Murtadha Hubail [ murtadha.hubail ]
            till Till Westmann made changes -
            Labels system-test system-test triaged
            murtadha.hubail Murtadha Hubail made changes -
            Rank Ranked higher
            murtadha.hubail Murtadha Hubail made changes -
            Sprint CX Sprint 207 [ 1145 ]
            murtadha.hubail Murtadha Hubail made changes -
            Rank Ranked lower
            murtadha.hubail Murtadha Hubail made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            till Till Westmann made changes -
            Remote Link This issue links to "*DB review (Web Link)" [ 20236 ]
            till Till Westmann made changes -
            Link This issue blocks MB-38724 [ MB-38724 ]
            till Till Westmann made changes -
            Labels system-test triaged approved-for-6.6.0 system-test triaged
            till Till Westmann made changes -
            Link This issue relates to MB-40472 [ MB-40472 ]
            Hide
            murtadha.hubail Murtadha Hubail added a comment -

            This issue happened because some rebalance was interrupted (e.g. cancelled or a crash happened in the middle of the rebalance) and that led to some unnecessary data existing from the cancelled rebalance. When the next rebalance came, we have a cleanup step that should've deleted the left over data. However, the cleanup process was not pointing to the correct file that should've been deleted because the file path wasn't resolved on the local node. This should be fixed now. 

            Show
            murtadha.hubail Murtadha Hubail added a comment - This issue happened because some rebalance was interrupted (e.g. cancelled or a crash happened in the middle of the rebalance) and that led to some unnecessary data existing from the cancelled rebalance. When the next rebalance came, we have a cleanup step that should've deleted the left over data. However, the cleanup process was not pointing to the correct file that should've been deleted because the file path wasn't resolved on the local node. This should be fixed now. 
            murtadha.hubail Murtadha Hubail made changes -
            Assignee Murtadha Hubail [ murtadha.hubail ] Ritam Sharma [ ritam.sharma ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            ritam.sharma Ritam Sharma made changes -
            VERIFICATION STEPS Verified and closed for 6.6.0-7883. System testing run did not show any exception for 3 days of test run.
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              ritam.sharma Ritam Sharma
              Reporter:
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty