Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45916

[windows]:FTS rebalance_out operation took > 1 hr to complete

    XMLWordPrintable

Details

    Description

      Build: Enterprise Edition 7.0.0 build 5014

      Scenario:

      1. 15 node cluster with couchbase bucket (1 replica)
      2. Deployed few fts functions
      3. Rebalance out eventing+fts node

        +----------------+---------------+-----------------------+----------------+--------------+
        | Nodes          | Services      | Version               | CPU            | Status       |
        +----------------+---------------+-----------------------+----------------+--------------+
        | 172.23.120.100 | kv            | 7.0.0-5014-enterprise | 17.6043133693  | Cluster node |
        | 172.23.136.114 | index, n1ql   | 7.0.0-5014-enterprise | 2.63076756414  | Cluster node |
        | 172.23.136.106 | index, n1ql   | 7.0.0-5014-enterprise | 36.5121957317  | Cluster node |
        | 172.23.136.107 | cbas          | 7.0.0-5014-enterprise | 1.49752495875  | Cluster node |
        | 172.23.120.113 | kv            | 7.0.0-5014-enterprise | 12.8772853786  | Cluster node |
        | 172.23.120.117 | kv            | 7.0.0-5014-enterprise | 12.8258333333  | Cluster node |
        | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 3.2941117648   | Cluster node |
        | 172.23.121.81  | index, n1ql   | 7.0.0-5014-enterprise | 13.6955706488  | Cluster node |
        | 172.23.120.144 | backup        | 7.0.0-5014-enterprise | 1.72978999567  | Cluster node |
        | 172.23.136.108 | cbas          | 7.0.0-5014-enterprise | 0.286669055575 | Cluster node |
        | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 4.43064872297  | Cluster node |
        | 172.23.136.115 | cbas          | 7.0.0-5014-enterprise | 2.05748285431  | Cluster node |
        | 172.23.136.113 | kv            | 7.0.0-5014-enterprise | 39.4535091082  | Cluster node |
        | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 3.41           | --- OUT ---> |
        | 172.23.136.105 | kv            | 7.0.0-5014-enterprise | 9.53166666667  | Cluster node |
        +----------------+---------------+-----------------------+----------------+--------------+
        

      Observation:

      Rebalance operation got stuck around 41.8%

      Starting rebalance, KeepNodes = ['ns_1@172.23.120.100','ns_1@172.23.136.114',
      'ns_1@172.23.136.106','ns_1@172.23.136.107',
      'ns_1@172.23.120.113','ns_1@172.23.120.117',
      'ns_1@172.23.138.127','ns_1@172.23.121.81',
      'ns_1@172.23.120.144','ns_1@172.23.136.108',
      'ns_1@172.23.136.112','ns_1@172.23.136.115',
      'ns_1@172.23.136.113','ns_1@172.23.136.105'], 
      EjectNodes = ['ns_1@172.23.136.110'], 
      Failed over and being ejected nodes = []; no delta recovery nodes;
      Operation Id = cea6ba07344c792eb6a229fa2bc04a6f

      Attachments

        Activity

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty