Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48513

Received error on DCP stream for vb: 868, err: document exists

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.1.0
    • 7.1.0
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1300
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

      • Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
      • Create 1 bucket, 100 scopes and 50 collections in each scopes
      • Created 500 indexes: one index (1 partition) on each collection
      • Create 2500 GSI indexes ( 5 on each collection)
      • Load documents on some of the collections
      • Run fts query workload
      • Kill fts on 172.23.106.242 and wait for 25 mins
      • Rebalance in 172.23.107.89 with fts service
      • Rebalance failed due OOM kill of fts node on 172.23.106.242
      • Kill fts on 172.23.106.243 and wait for 15 mins
      • Mutate and add some more docs
      • rebalance out 172.23.106.242

      Seeing below errors in 172.23.106.242:

      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-09-17T16:34:19.820-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_dpdue-type-country-price-public_likes-Overall-free_parking_44ee2d02fd7d2370_4c1c5584] Received error on DCP stream for vb: 868, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":3088,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.106.242:56120","last_connection_id":"1565eaefab6bf6d2/1fa75307d3260e7a"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-09-17T16:34:19.821-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_dpdue-type-country-price-public_likes-Overall-free_parking_44ee2d02fd7d2370_4c1c5584] Received error on DCP stream for vb: 872, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":3089,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.106.242:56120","last_connection_id":"1565eaefab6bf6d2/1fa75307d3260e7a"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-09-17T16:34:19.825-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_dpdue-type-country-price-public_likes-Overall-free_parking_44ee2d02fd7d2370_4c1c5584] Received error on DCP stream for vb: 882, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":3090,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.106.242:56120","last_connection_id":"1565eaefab6bf6d2/1fa75307d3260e7a"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-09-17T16:34:19.825-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_dpdue-type-country-price-public_likes-Overall-free_parking_44ee2d02fd7d2370_4c1c5584] Received error on DCP stream for vb: 887, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":3091,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.106.242:56120","last_connection_id":"1565eaefab6bf6d2/1fa75307d3260e7a"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964

      In 172.23.107.89:

      2021-09-17T17:10:00.881-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 171, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2489,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      2021-09-17T17:10:00.881-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 174, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2490,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      2021-09-17T17:10:00.881-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 177, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2491,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      2021-09-17T17:10:00.881-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 172, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2492,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      2021-09-17T17:10:00.885-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 173, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2493,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      2021-09-17T17:10:00.885-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 179, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2494,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964
      		 
      2021-09-17T17:10:00.885-07:00 [ERRO] feed_dcp_gocbcore: [bucket_bucket1_idx_xpwqj-Overall-public_likes-type-free_parking_1273988a6fef1ab5_4c1c5584] Received error on DCP stream for vb: 176, err: document exists | {"status_code":2,"bucket":"bucket1","error_name":"KEY_EEXISTS","error_description":"key already exists, or CAS mismatch","opaque":2495,"last_dispatched_to":"172.23.106.253:11207","last_dispatched_from":"172.23.107.89:40838","last_connection_id":"47cb49fd9ae79eaa/02d40b01622bb148"} -- cbgt.(*GocbcoreDCPFeed).initiateStreamEx.func1() at feed_dcp_gocbcore.go:964

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631924630/collectinfo-2021-09-18T002351-ns_1%40172.23.106.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631924630/collectinfo-2021-09-18T002351-ns_1%40172.23.106.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631924630/collectinfo-2021-09-18T002351-ns_1%40172.23.106.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1631924630/collectinfo-2021-09-18T002351-ns_1%40172.23.107.89.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-51278 [System test upgrade] - Received error on DCP stream for vb: 272, err: document exists

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhinav Abhi Dangeti
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty