Loading...

Details

Type: Bug
Resolution: Cannot Reproduce
Priority: Critical
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: secondary-index
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.106.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.151.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.74.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.106.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.106.136.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.58.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.73.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.74.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.81.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.120.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.121.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.24.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.26.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.31.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.123.33.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.14.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.243.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.254.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.96.48.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.105.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.110.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.149.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.150.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.151.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.241.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618968544/collectinfo-2021-04-21T012906-ns_1%40172.23.97.74.zip
Story Points:
1
Is this a Regression?:
No

Description

Build : 7.0.0-4960
Test : -test tests/2i/cheshirecat/test_idx_cc_vol_10K_moi_tmp.yml -scope tests/2i/cheshirecat/scope_idx_cc_vol_10K_moi.yml (MOI 10K indexes test)
Scale : 5

In the volume test, the cluster ops steps were restarted after setting the settings :
indexer.planner.timeout : 14400 (4 hrs)
indexer.planner.variationThreshold : 0.1

NS server inactivity timeout for indexer also set to 4 hrs.

The first rebalance operation looks like it is hung as it is going on for 20+ hrs. This seems different than ~~MB-45461~~ as currently there are no index movements happening. All indexes are in Ready state as seen on the getIndexStatus output (attached).

Indexer nodes : 172.23.106.136, 172.23.120.58, 172.23.120.74, 172.23.120.75, 172.23.120.77, 172.23.120.81, 172.23.120.86, 172.23.123.31, 172.23.123.32, 172.23.123.33, 172.23.96.243, 172.23.96.254, 172.23.97.105, 172.23.97.110, 172.23.97.112

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

106.136_getIndexStatus.json
6.64 MB
20/Apr/21 7:41 PM

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[Volume Test] : Indexer rebalance stuck

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty