Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49592

Rebalance out fts node is stuck at 33%

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1695

      • 3 node Cluster - kv, fts, fts
      • Create a Bucket and Load 30K docs
      • Create below fts index upside_down indexes and wait for them to complete

      [2021-11-16 20:05:04,349] - [fts_base:1230] INFO - Creating fulltext-index default_index_1 on 172.23.108.116
      [2021-11-16 20:05:04,349] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "default_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
       
       
      [2021-11-16 20:05:05,289] - [fts_base:1230] INFO - Creating fulltext-index standard_bucket_1_index_1 on 172.23.108.116
      [2021-11-16 20:05:05,289] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "standard_bucket_1_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "standard_bucket_1", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
       
      [2021-11-16 20:05:05,856] - [fts_base:1230] INFO - Creating fulltext-index standard_bucket_2_index_1 on 172.23.108.116
      [2021-11-16 20:05:05,857] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "standard_bucket_2_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "standard_bucket_2", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
       
       
      [2021-11-16 20:05:11,383] - [fts_base:1230] INFO - Creating fulltext-index standard_bucket_3_index_1 on 172.23.108.116
      [2021-11-16 20:05:11,383] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "standard_bucket_3_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "standard_bucket_3", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
      

      Rebalnce out FTS node (172.23.108.117). This hangs at 33%

      [2021-11-16 20:06:27,062] - [fts_base:3264] INFO - Starting rebalance-out nodes:[ip:172.23.108.117 port:8091 ssh_username:root] at C1 cluster 172.23.108.115
      [2021-11-16 20:06:27,830] - [rest_client:1843] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.108.115,ns_1@172.23.108.116,ns_1@172.23.108.117', 'ejectedNodes': 'ns_1@172.23.108.117', 'user': 'Administrator', 'password': 'password'}
      [2021-11-16 20:06:27,894] - [rest_client:1848] INFO - rebalance operation started
      [2021-11-16 20:06:37,911] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
      [2021-11-16 20:06:37,911] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
      [2021-11-16 20:06:57,956] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
      [2021-11-16 20:06:57,956] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
      [2021-11-16 20:07:18,000] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
      [2021-11-16 20:07:18,001] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
      [2021-11-16 20:07:38,049] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
      [2021-11-16 20:07:38,049] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
      [2021-11-16 20:07:58,096] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
      [2021-11-16 20:07:58,096] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
      [2021-11-16 20:08:18,143] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
      [2021-11-16 20:08:18,144] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
      

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-11-17T042256-ns_1%40172.23.108.115.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-11-17T042256-ns_1%40172.23.108.116.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-11-17T042256-ns_1%40172.23.108.117.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty