Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36490

Unable to add back node which was rebalanced out

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-4558

      Test: MH longevity

      Cycle: 1st 

      Day: 1st 

      We remove node .135 

      [2019-10-14T23:29:55-07:00, sequoiatools/couchbase-cli:6.5:2fc1ad] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2019-10-14T23:30:10-07:00, sequoiatools/couchbase-cli:6.5:213ce0] rebalance -c 172.23.108.103:8091 --server-remove 172.23.98.135 -u Administrator -p password
      [2019-10-14T23:37:56-07:00, sequoiatools/cmd:7189ca] 60
      

      Start and completed logs for the rebalance as it completed successfully 

      160023:[user:info,2019-10-14T23:30:12.305-07:00,ns_1@172.23.108.103:<0.10652.0>:ns_orchestrator:idle:828]Starting rebalance, KeepNodes = ['ns_1@172.23.104.156','ns_1@172.23.104.164',
                                   'ns_1@172.23.104.5','ns_1@172.23.104.61',
                              'ns_1@172.23.104.67','ns_1@172.23.104.69',
                                      'ns_1@172.23.104.70','ns_1@172.23.104.87',
                                     'ns_1@172.23.104.88','ns_1@172.23.106.100',
                                     'ns_1@172.23.106.188','ns_1@172.23.108.103',
                                     'ns_1@172.23.96.148','ns_1@172.23.96.251',
                                   'ns_1@172.23.96.252','ns_1@172.23.96.253',
                                    'ns_1@172.23.96.95','ns_1@172.23.97.119',
                                  'ns_1@172.23.97.121','ns_1@172.23.97.122',
                                  'ns_1@172.23.97.239','ns_1@172.23.97.242',
                                       'ns_1@172.23.99.11','ns_1@172.23.99.20',
                                      'ns_1@172.23.99.21','ns_1@172.23.99.25'], EjectNodes = ['ns_1@172.23.98.135'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 5841804a715ebcac0a55e92c5d7be282 

      and completed successfully 

      [user:info,2019-10-14T23:37:47.494-07:00,ns_1@172.23.108.103:<0.10652.0>:ns_orchestrator:log_rebalance_completion:1466]Rebalance completed successfully.
      Rebalance Operation Id = 5841804a715ebcac0a55e92c5d7be282 

      Then try to add back but fails as its already part of the cluster even though its not part of cluster

      [2019-10-14T23:44:29-07:00, sequoiatools/couchbase-cli:6.5:5b3b76] server-add -c 172.23.108.103:8091 --server-add https://172.23.98.135 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[server-add -c 172.23.108.103:8091 --server-add https://172.23.98.135 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing]
       
      docker logs 5b3b76
      docker start 5b3b76
       
      =ERROR: Prepare join failed. Node is already part of cluster.
      [2019-10-14T23:44:38-07:00, sequoiatools/couchbase-cli:6.5:701f5a] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      [2019-10-14T23:45:38-07:00, sequoiatools/cmd:846e63] 60 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              trond Trond Norbye
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty