Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.0.2
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.2-6522
Test: -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale: 3
Seeing below errors and then we see fts crashed in the node 172.23.106.134
172.23.106.134 : fts
|
2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 316, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":722,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
|
2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 317, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":723,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
|
2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 318, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":724,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
|
2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 319, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":725,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
|
2021-08-17T20:51:18.672-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 338, err: document exists |
|
|
|
|
|
|
172.23.106.134 : crash
|
[user:info,2021-08-17T20:51:49.751-07:00,ns_1@172.23.106.134:<0.20048.0>:ns_log:crash_consumption_loop:63]Service 'fts' exited with status 137. Restarting. Messages:
|
At this point test was at this step:
[2021-08-17T20:43:28-07:00, sequoiatools/couchbase-cli:7.0:67d1a4] server-add -c 172.23.97.74:8091 --server-add https://172.23.97.112 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2021-08-17T20:44:20-07:00, sequoiatools/couchbase-cli:7.0:f18f24] failover -c 172.23.97.74:8091 --server-failover 172.23.96.14:8091 -u Administrator -p password --hard
|
[2021-08-17T20:44:33-07:00, sequoiatools/couchbase-cli:7.0:990892] rebalance -c 172.23.97.74:8091 -u Administrator -p password
|
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.97.74:8091 -u Administrator -p password]
|
|
docker logs 990892
|
docker start 990892
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2021-08-17T20:48:24-07:00, sequoiatools/cmd:ce9933] 60
|
*[2021-08-17T20:49:31-07:00, sequoiatools/cmd:137f03] 600*
|
[2021-08-17T20:59:38-07:00, appropriate/curl:a3b954] -u Administrator:password -X POST http://172.23.97.74:8091/settings/replications/fed297b51791741803659bfad2a59818/bucket8/bucket8 -d pauseRequested=true
|
[2021-08-17T20:59:46-07:00, sequoiatools/cmd:1484da] 300
|
We see the swap rebalance data node failed with mover_crashed error and then later in fts node, we see KEY_EXISTS error and FTS crash.
Cluster config:
########## Cluster config ##################
|
###### cbas : 3 ===== > [172.23.120.58:8091 172.23.120.74:8091 172.23.120.75:8091] ###########
|
###### kv : 11 ===== > [172.23.120.73:8091 172.23.120.77:8091 172.23.120.86:8091 172.23.121.77:8091 172.23.123.24:8091 172.23.123.25:8091 172.23.123.26:8091 172.23.96.122:8091 172.23.96.14:8091 172.23.97.241:8091 172.23.97.74:8091] ###########
|
###### index : 6 ===== > [172.23.120.81:8091 172.23.96.243:8091 172.23.96.254:8091 172.23.97.105:8091 172.23.97.110:8091 172.23.97.148:8091] ###########
|
###### backup : 1 ===== > [172.23.123.32:8091] ###########
|
###### n1ql : 2 ===== > [172.23.97.149:8091 172.23.97.150:8091] ###########
|
###### fts : 2 ===== > [172.23.106.134:8091 172.23.97.151:8091] ###########
|
###### eventing : 4 ===== > [172.23.106.136:8091 172.23.123.31:8091 172.23.123.33:8091 172.23.96.48:8091] ###########
|
Logs:
cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.74.zip