Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4902

status stopped by janitor error message during Rebalance

    XMLWordPrintable

Details

    Description

      Scenario:
      1) 30 nodes cluster
      2) Add and remove 18 nodes at the same time (+18 -18)
      3) The 4 nodes, are still pending add, even after rebalance was completed.

      10.224.111.34
      10.224.111.99
      10.224.113.96
      10.224.114.133

      http://ec2-50-17-160-100.compute-1.amazonaws.com:8091/index.html#sec=servers&settingsTabs=1&serversTab=1

      Getting teh diags, will attach them here.

      From UI:-

      Conflicting configuration changes to field nodes_wanted:
      [{'_vclock',[{'ns_1@10.101.42.239',{62,63499073848}},
      {'ns_1@10.101.45.91',{31,63499073846}},
      {'ns_1@10.114.225.102',{18,63499062792}},
      {'ns_1@10.114.226.155',{12,63499061481}},
      {'ns_1@10.114.226.220',{18,63499062638}},
      {'ns_1@10.224.109.59',{30,63499062748}},
      {'ns_1@10.224.110.233',{65,63499061385}},
      {'ns_1@10.224.117.247',{1,63499067192}},
      {'ns_1@10.224.118.136',{1,63499067192}},
      {'ns_1@10.224.118.214',{1,63499067192}},
      {'ns_1@10.99.17.8',{1,63499067192}},
      {'ns_1@10.99.19.107',{148,63499073689}}]},
      'ns_1@10.101.42.239','ns_1@10.114.206.74','ns_1@10.114.225.188',
      'ns_1@10.114.226.155','ns_1@10.114.227.121','ns_1@10.114.227.165',
      'ns_1@10.114.33.152','ns_1@10.114.33.231','ns_1@10.116.117.245',
      'ns_1@10.224.109.186','ns_1@10.224.109.59','ns_1@10.224.109.91',
      'ns_1@10.224.110.164','ns_1@10.224.110.174','ns_1@10.224.110.175',
      'ns_1@10.224.110.185','ns_1@10.224.111.24','ns_1@10.224.111.34',
      'ns_1@10.224.111.99','ns_1@10.224.113.59','ns_1@10.224.113.96',
      'ns_1@10.224.114.133','ns_1@10.224.115.136','ns_1@10.224.115.50',
      'ns_1@10.224.117.243','ns_1@10.224.117.247','ns_1@10.224.117.82',
      'ns_1@10.224.118.136','ns_1@10.224.118.214','ns_1@10.224.119.138',
      'ns_1@10.224.119.44','ns_1@10.224.123.84','ns_1@10.99.17.8',
      'ns_1@10.99.19.107'] and
      [{'_vclock',[{'ns_1@10.101.42.239',{61,63499073534}},
      {'ns_1@10.101.45.91',{35,63499073846}},
      {'ns_1@10.114.225.102',{18,63499062792}},
      {'ns_1@10.114.226.155',{12,63499061481}},
      {'ns_1@10.114.226.220',{18,63499062638}},
      {'ns_1@10.224.109.59',{30,63499062748}},
      {'ns_1@10.224.110.233',{65,63499061385}},
      {'ns_1@10.224.117.247',{1,63499067192}},
      {'ns_1@10.224.118.136',{1,63499067192}},
      {'ns_1@10.224.118.214',{1,63499067192}},
      {'ns_1@10.99.17.8',{1,63499067192}},
      {'ns_1@10.99.19.107',{148,63499073689}}]},
      'ns_1@10.101.42.239','ns_1@10.101.45.91','ns_1@10.114.206.74',
      'ns_1@10.114.225.188','ns_1@10.114.226.155','ns_1@10.114.227.121',
      'ns_1@10.114.227.165','ns_1@10.114.33.152','ns_1@10.114.33.231',
      'ns_1@10.116.117.245','ns_1@10.224.109.186','ns_1@10.224.109.59',
      'ns_1@10.224.109.91','ns_1@10.224.110.164','ns_1@10.224.110.174',
      'ns_1@10.224.110.175','ns_1@10.224.110.185','ns_1@10.224.111.24',
      'ns_1@10.224.113.59','ns_1@10.224.115.136','ns_1@10.224.115.50',
      'ns_1@10.224.117.243','ns_1@10.224.117.247','ns_1@10.224.117.82',
      'ns_1@10.224.118.136','ns_1@10.224.118.214','ns_1@10.224.119.138',
      'ns_1@10.224.119.44','ns_1@10.224.123.84','ns_1@10.99.17.8',
      'ns_1@10.99.19.107'], choosing the former.

      Attachments

        1. 10.114.206.74-8091-diag.txt.gz
          3.03 MB
          Karan Kumar
        2. 10.114.227.165-8091-diag.txt.gz
          3.07 MB
          Karan Kumar
        3. 10.224.110.174-8091-diag.txt.gz
          3.02 MB
          Karan Kumar
        4. 10.224.110.175-8091-diag.txt.gz
          3.03 MB
          Karan Kumar
        5. 10.224.113.59-8091-diag.txt.gz
          3.16 MB
          Karan Kumar
        6. 10.224.115.136-8091-diag.txt.gz
          2.86 MB
          Karan Kumar
        7. 10.224.115.50-8091-diag.txt.gz
          2.93 MB
          Karan Kumar
        8. 10.224.117.243-8091-diag.txt.gz
          2.98 MB
          Karan Kumar
        9. 10.224.117.247-8091-diag.txt.gz
          2.94 MB
          Karan Kumar
        10. 10.224.118.136-8091-diag.txt.gz
          3.00 MB
          Karan Kumar
        11. 10.224.118.214-8091-diag.txt.gz
          2.73 MB
          Karan Kumar
        12. 10.224.119.138-8091-diag.txt.gz
          3.12 MB
          Karan Kumar
        13. 10.224.119.44-8091-diag.txt.gz
          3.15 MB
          Karan Kumar
        14. 10.224.123.84-8091-diag.txt.gz
          2.73 MB
          Karan Kumar
        15. 10.99.17.8-8091-diag.txt.gz
          3.02 MB
          Karan Kumar
        16. 10.99.17.8-8091-diag.txt.gz
          3.02 MB
          Karan Kumar
        17. 10.99.19.107-8091-diag.txt.gz
          3.98 MB
          Karan Kumar
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            farshid Farshid Ghods (Inactive)
            karan Karan Kumar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty