Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Untriaged
-
1
-
Unknown
Description
Build: 7.1.0-1695
- 3 node Cluster - kv, fts, fts
- Create a Bucket and Load 30K docs
- Create below fts index upside_down indexes and wait for them to complete
[2021-11-16 20:05:04,349] - [fts_base:1230] INFO - Creating fulltext-index default_index_1 on 172.23.108.116
|
[2021-11-16 20:05:04,349] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "default_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
|
|
|
[2021-11-16 20:05:05,289] - [fts_base:1230] INFO - Creating fulltext-index standard_bucket_1_index_1 on 172.23.108.116
|
[2021-11-16 20:05:05,289] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "standard_bucket_1_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "standard_bucket_1", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
|
|
[2021-11-16 20:05:05,856] - [fts_base:1230] INFO - Creating fulltext-index standard_bucket_2_index_1 on 172.23.108.116
|
[2021-11-16 20:05:05,857] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "standard_bucket_2_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "standard_bucket_2", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
|
|
|
[2021-11-16 20:05:11,383] - [fts_base:1230] INFO - Creating fulltext-index standard_bucket_3_index_1 on 172.23.108.116
|
[2021-11-16 20:05:11,383] - [rest_client:3457] INFO - {"type": "fulltext-index", "name": "standard_bucket_3_index_1", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "upside_down"}}, "sourceType": "couchbase", "sourceName": "standard_bucket_3", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}}
|
Rebalnce out FTS node (172.23.108.117). This hangs at 33%
[2021-11-16 20:06:27,062] - [fts_base:3264] INFO - Starting rebalance-out nodes:[ip:172.23.108.117 port:8091 ssh_username:root] at C1 cluster 172.23.108.115
|
[2021-11-16 20:06:27,830] - [rest_client:1843] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.108.115,ns_1@172.23.108.116,ns_1@172.23.108.117', 'ejectedNodes': 'ns_1@172.23.108.117', 'user': 'Administrator', 'password': 'password'}
|
[2021-11-16 20:06:27,894] - [rest_client:1848] INFO - rebalance operation started
|
[2021-11-16 20:06:37,911] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
|
[2021-11-16 20:06:37,911] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
|
[2021-11-16 20:06:57,956] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
|
[2021-11-16 20:06:57,956] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
|
[2021-11-16 20:07:18,000] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
|
[2021-11-16 20:07:18,001] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
|
[2021-11-16 20:07:38,049] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
|
[2021-11-16 20:07:38,049] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
|
[2021-11-16 20:07:58,096] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
|
[2021-11-16 20:07:58,096] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
|
[2021-11-16 20:08:18,143] - [rest_client:2010] INFO - rebalance percentage : 33.00 %
|
[2021-11-16 20:08:18,144] - [task:841] INFO - Rebalance - status: running, progress: 33.00%
|
Logs:
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-11-17T042256-ns_1%40172.23.108.115.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-11-17T042256-ns_1%40172.23.108.116.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-11-17T042256-ns_1%40172.23.108.117.zip