Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46154

FTS rebalance_out stuck for >1hr

    XMLWordPrintable

Details

    Description

      Build: 7.0.0-5112

      Snapshot: http://supportal.couchbase.com/snapshot/3894907ec0587b4d955a6fdb427c8ee9::0

      Rebalance out FTS node (172.23.107.44) got stuck.

      +----------------+-------------+-----------------------+----------------+--------------+
      | Nodes          | Services    | Version               | CPU            | Status       |
      +----------------+-------------+-----------------------+----------------+--------------+
      | 172.23.107.58  | eventing    | 7.0.0-5112-enterprise | 64.2933333333  | Cluster node |
      | 172.23.107.44  | fts         | 7.0.0-5112-enterprise | 1.18045962577  | --- OUT ---> |
      | 172.23.105.175 | kv          | 7.0.0-5112-enterprise | 84.6334463866  | Cluster node |
      | 172.23.107.45  | fts         | 7.0.0-5112-enterprise | 2.16461112509  | Cluster node |
      | 172.23.107.47  | kv          | 7.0.0-5112-enterprise | 84.0340537001  | Cluster node |
      | 172.23.107.84  | eventing    | 7.0.0-5112-enterprise | 61.2253016841  | Cluster node |
      | 172.23.121.78  | cbas        | 7.0.0-5112-enterprise | 32.9093586826  | Cluster node |
      | 172.23.106.250 | index, n1ql | 7.0.0-5112-enterprise | 0.626723489596 | Cluster node |
      | 172.23.107.88  | index, n1ql | 7.0.0-5112-enterprise | 0.57738169951  | Cluster node |
      | 172.23.107.43  | eventing    | 7.0.0-5112-enterprise | 67.6125115481  | Cluster node |
      | 172.23.106.236 | kv          | 7.0.0-5112-enterprise | 84.7989623865  | Cluster node |
      | 172.23.107.85  | index, n1ql | 7.0.0-5112-enterprise | 22.0259128386  | Cluster node |
      | 172.23.106.251 | backup      | 7.0.0-5112-enterprise | 0.690781210751 | Cluster node |
      | 172.23.106.233 | kv          | 7.0.0-5112-enterprise | 86.3858835652  | Cluster node |
      | 172.23.106.238 | index, n1ql | 7.0.0-5112-enterprise | 22.4690360797  | Cluster node |
      | 172.23.107.54  | kv          | 7.0.0-5112-enterprise | 83.7830825117  | Cluster node |
      | 172.23.107.78  | cbas        | 7.0.0-5112-enterprise | 29.1700241741  | Cluster node |
      | 172.23.121.74  | cbas        | 7.0.0-5112-enterprise | 31.6915623738  | Cluster node |
      | 172.23.107.107 | fts         | 7.0.0-5112-enterprise | 0.614882670348 | Cluster node |
      | 172.23.107.91  | kv          | 7.0.0-5112-enterprise | 88.9574495436  | Cluster node |
      +----------------+-------------+-----------------------+----------------+--------------+

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Update: Rebalance completed after 1hr 15mins.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Update : Rebalance completed after 1hr 15mins.

            After checking the logs, I don't see any other errors here. The only issue is the low RAM quota configured (512MB) which is really slowing the rebalance operations here.

            Closing the ticket.

            Sreekanth Sivasankaran Sreekanth Sivasankaran added a comment - After checking the logs, I don't see any other errors here. The only issue is the low RAM quota configured (512MB) which is really slowing the rebalance operations here. Closing the ticket.

            Updated the memory settings to 2GB and not seeing this issue again.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Updated the memory settings to 2GB and not seeing this issue again.

            People

              Sreekanth Sivasankaran Sreekanth Sivasankaran
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty