Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: secondary-index
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.18.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.151.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.74.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.58.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.73.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.74.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.81.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.120.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.121.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.24.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.26.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.31.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.123.33.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.14.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.18.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.243.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.254.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.96.48.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.105.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.110.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.149.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.150.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.151.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1600582278/collectinfo-2020-09-20T061119-ns_1%40172.23.97.74.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build :
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Day : 1st (1st iteration)

This is the longevity test with Mad-hatter (no collection support). The test started at 2020-09-18T12:34. Then at 2020-09-19T15:17, there was a step to add a new index node to the cluster - 172.23.120.81.

[2020-09-19T15:17:23-07:00, sequoiatools/couchbase-cli:7.0:24b7a5] server-add -c 172.23.97.74:8091 --server-add https://172.23.120.81 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index

[2020-09-19T15:17:36-07:00, sequoiatools/couchbase-cli:7.0:0ef535] rebalance -c 172.23.97.74:8091 -u Administrator -p password

Index default_result_ratings_claims_pindex which is a partitioned index with replica began replicating across the following nodes - 172.23.123.32, 172.23.97.112, 172.23.97.105, and 172.23.120.81 (new node). The index is in Ready state on the new node 172.23.120.81, but still stuck in Replicating state on the remaining nodes. It has been 65 hours since then.

This is holding up the rebalance operation, and also the test cannot move forward.

Logs attached are from ~9/19 11 PM PST. If you need logs from a later time, pls let me know.