Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48023

[FTS] Service 'fts' exited with status 137 - Received error on DCP stream for vb: 190, err: document exists

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.2-6522
      Test: -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale: 3

      Seeing below errors and then we see fts crashed in the node 172.23.106.134

      172.23.106.134 : fts
          2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 316, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":722,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
          2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 317, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":723,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
          2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 318, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":724,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
          2021-08-17T20:51:18.477-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 319, err: document exists | {"status_code":2,"bucket":"default","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":725,"last_dispatched_to":"172.23.123.24:11207","last_dispatched_from":"172.23.106.134:33574","last_connection_id":"f22009344f941886/7d3cb6fb6db64c19"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:963
          2021-08-17T20:51:18.672-07:00 [ERRO] feed_dcp_gocbcore: [social_54b9f78e1ac75c89_f4e0a48a] Received error on DCP stream for vb: 338, err: document exists | 
       
       
       
       
       
          172.23.106.134 : crash
          [user:info,2021-08-17T20:51:49.751-07:00,ns_1@172.23.106.134:<0.20048.0>:ns_log:crash_consumption_loop:63]Service 'fts' exited with status 137. Restarting. Messages:
      

      At this point test was at this step:

      [2021-08-17T20:43:28-07:00, sequoiatools/couchbase-cli:7.0:67d1a4] server-add -c 172.23.97.74:8091 --server-add https://172.23.97.112 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2021-08-17T20:44:20-07:00, sequoiatools/couchbase-cli:7.0:f18f24] failover -c 172.23.97.74:8091 --server-failover 172.23.96.14:8091 -u Administrator -p password --hard
      [2021-08-17T20:44:33-07:00, sequoiatools/couchbase-cli:7.0:990892] rebalance -c 172.23.97.74:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.97.74:8091 -u Administrator -p password]
       
      docker logs 990892
      docker start 990892
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2021-08-17T20:48:24-07:00, sequoiatools/cmd:ce9933] 60
      *[2021-08-17T20:49:31-07:00, sequoiatools/cmd:137f03] 600*
      [2021-08-17T20:59:38-07:00, appropriate/curl:a3b954] -u Administrator:password -X POST http://172.23.97.74:8091/settings/replications/fed297b51791741803659bfad2a59818/bucket8/bucket8 -d pauseRequested=true
      [2021-08-17T20:59:46-07:00, sequoiatools/cmd:1484da] 300
      

      We see the swap rebalance data node failed with mover_crashed error and then later in fts node, we see KEY_EXISTS error and FTS crash.

      Cluster config:

      ########## Cluster config ##################
      ######  cbas : 3 ===== > [172.23.120.58:8091 172.23.120.74:8091 172.23.120.75:8091]  ###########
      ######  kv : 11 ===== > [172.23.120.73:8091 172.23.120.77:8091 172.23.120.86:8091 172.23.121.77:8091 172.23.123.24:8091 172.23.123.25:8091 172.23.123.26:8091 172.23.96.122:8091 172.23.96.14:8091 172.23.97.241:8091 172.23.97.74:8091]  ###########
      ######  index : 6 ===== > [172.23.120.81:8091 172.23.96.243:8091 172.23.96.254:8091 172.23.97.105:8091 172.23.97.110:8091 172.23.97.148:8091]  ###########
      ######  backup : 1 ===== > [172.23.123.32:8091]  ###########
      ######  n1ql : 2 ===== > [172.23.97.149:8091 172.23.97.150:8091]  ###########
      ######  fts : 2 ===== > [172.23.106.134:8091 172.23.97.151:8091]  ###########
      ######  eventing : 4 ===== > [172.23.106.136:8091 172.23.123.31:8091 172.23.123.33:8091 172.23.96.48:8091]  ###########
      

      Logs:

      cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.136.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.149.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.74.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty