Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: secondary-index
Labels:
- system-test

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.25.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.155.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.157.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.67.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.69.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.104.70.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.105.107.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.106.188.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.108.103.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.120.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.123.27.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.123.28.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.251.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.252.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.96.253.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.97.119.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.97.121.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.98.135.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.11.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.20.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.21.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613932046/collectinfo-2021-02-21T182728-ns_1%40172.23.99.25.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.0.0-4495
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Iteration : 2nd
Day : 2nd
Scale : 2

In the longevity system test, there is a step to remove an indexer node 172.23.104.67.

[2021-02-21T02:41:53-08:00, sequoiatools/couchbase-cli:7.0:71fada] rebalance -c 172.23.108.103:8091 --server-remove 172.23.104.67 -u Administrator -p password

This rebalance operation seems to be stuck as it has not yet completed since 37+ hrs.

From the get_index_status, there are some indexes that are in Error state, and some in Moving state. Attached is the output of getIndexStatus rest endpoint.

The logs attached were collected at around 2021-02-21 10:30 AM. If you need logs from before or after this, please let me know.

Indexer nodes in the cluster :
172.23.96.253, 172.23.96.252, 172.23.104.69, 172.23.104.67