Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48699

[System Test] DELETE queries executed through eventing handlers failed with error NOT_MY_VBUCKET

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1377
      Test : -test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
      Scale : 3
      Iteration : 1st

      There are 12 DELETE queries executed through eventing handlers that failed due to reason - Some keys were not deleted [<key id>]
      Example completed request :

      {
                  "completed_requests": {
                      "node": "172.23.120.245:8091",
                      "users": "@eventing",
                      "state": "completed",
                      "errors": [
                          {
                              "code": 12011,
                              "icause": "MCResponse status=NOT_MY_VBUCKET, opcode=DELETE, opaque=2, msg: {\"rev\":61627,\"revEpoch\":1,\"name\":\"NEW_ORDER\",\"nodeLocator\":\"vbucket\",\"bucketType\":\"membase\",\"uuid\":\"bb98642244743421670c71047afa257a\",\"uri\":\"/pools/default/buckets/NEW_ORDER?bucket_uuid=bb98642244743421670c71047afa257a\",\"streamingUri\":\"/pools/default/bucketsStreaming/NEW_ORDER?bucket_uuid=bb98642244743421670c71047afa257a\",\"bucketCapabilitiesVer\":\"\",\"bucketCapabilities\":[\"collections\",\"durableWrite\",\"tombstonedUserXAttrs\",\"couchapi\",\"subdoc.ReplaceBodyWithXattr\",\"subdoc.DocumentMacroSupport\",\"dcp\",\"cbhello\",\"touch\",\"cccp\",\"xdcrCheckpointing\",\"nodesExt\",\"xattr\"],\"collectionsManifestUid\":\"5\",\"ddocs\":{\"uri\":\"/pools/default/buckets/NEW_ORDER/ddocs\"},\"vBucketServerMap\":{\"hashAlgorithm\":\"CRC\",\"numReplicas\":1,\"serverList\":[\"172.23.104.5:11210\",\"172.23.104.69:11210\",\"172.23.108.103:11210\",\"172.23.96.148:11210\",\"172.23.97.119:11210\",\"172.23.97.121:11210\",\"172.23.97.122:11210\",\"172.23.97.239:11210\",\"172.23.99.20:11210\",\"172.23.99.21:11210\",\"172.23.99.25:11210\"],\"vBucketMap\":[[0,5],[0,6],[0,7],[0,8],[0,9],[0,10],[0,10],[0,10],[0,10],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,8],[0,8],[0,8],[0,8],[0,8],[0,8],[0,8],[0,8],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[2,0],[2,3],[2,4],[2,5],[2,7],[2,-1],[2,8],[0,2],[0,3],[3,0],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[4,2],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[5,0],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[6,0],[6,3],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[7,0],[7,2],[7,4],[7,5],[7,6],[7,6],[7,6],[7,6],[0,-1],[2,-1],[3,-1],[4,-1],[6,-1],[1,7],[1,7],[1,7],[8,2],[8,3],[8,4],[8,5],[8,-1],[8,-1],[8,-1],[8,-1],[8,-1],[9,0],[9,2],[9,3],[9,4],[9,5],[9,6],[9,8],[9,8],[10,0],[10,3],[10,4],[10,5],[10,6],[10,7],[10,8],[10,9],[10,9],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,7],[2,7],[2,7],[2,7],[2,7],[2,7],[2,7],[2,7],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,8],[2,8],[2,8],[2,8],[2,8],[2,8],[2,8],[2,8],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[0,10],[0,10],[0,10],[0,10],[0,10],[0,10],[3,2],[3,2],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,8],[3,8],[3,8],[3,8],[3,8],[3,8],[3,8],[3,8],[3,9],[3,9],[3,9],[3,9],[3,9],[3,9],[3,9],[3,9],[3,10],[3,10],[3,10],[3,10],[3,10],[3,10],[3,10],[3,10],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[3,-1],[3,8],[3,9],[3,10],[4,3],[4,3],[4,5],[4,7],[4,-1],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,7],[4,7],[4,7],[4,7],[4,7],[4,7],[4,7],[4,7],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,8],[4,8],[4,8],[4,8],[4,8],[4,8],[4,8],[4,8],[4,9],[4,9],[4,9],[4,9],[4,9],[4,9],[4,9],[4,9],[4,10],[4,10],[4,10],[4,10],[4,10],[4,10],[4,10],[4,10],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[4,8],[4,9],[4,10],[5,4],[5,4],[5,6],[5,7],[5,-1],[5,8],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,8],[5,8],[5,8],[5,8],[5,8],[5,8],[5,8],[5,8],[5,9],[5,9],[5,9],[5,9],[5,9],[5,9],[5,9],[5,9],[5,10],[5,10],[5,10],[5,10],[5,10],[5,10],[5,10],[5,10],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[5,9],[5,10],[6,5],[6,5],[6,5],[6,5],[6,7],[6,-1],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,-1],[6,-1],[6,-1],[6,-1],[6,-1],[6,-1],[6,-1],[6,-1],[6,-1],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,9],[6,9],[6,9],[6,9],[6,9],[6,9],[6,9],[6,9],[6,10],[6,10],[6,10],[6,10],[6,10],[6,10],[6,10],[6,10],[7,0],[7,0],[7,0],[7,0],[7,0],[7,0],[7,0],[7,0],[6,9],[6,10],[7,6],[7,6],[7,6],[7,6],[7,6],[7,6],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,4],[7,4],[7,4],[7,4],[7,4],[7,4],[7,4],[7,4],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,-1],[7,-1],[7,-1],[7,-1],[7,-1],[7,-1],[7,-1],[7,-1],[7,-1],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,10],[7,10],[7,10],[7,10],[7,10],[7,10],[7,10],[7,10],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[7,-1],[7,10],[1,7],[1,7],[1,7],[1,7],[1,7],[1,7],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[2,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[3,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[4,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[5,-1],[6,-1],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[8,-1],[8,-1],[8,-1],[8,-1],[8,-1],[8,-1],[8,-1],[8,-1],[1,8],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[1,9],[10,-1],[10,-1],[10,-1],[10,-1],[1,10],[1,10],[1,10],[1,10],[1,10],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[1,7],[1,8],[8,-1],[8,-1],[8,-1],[8,-1],[8,-1],[8,9],[8,2],[8,2],[8,2],[8,2],[8,2],[8,2],[8,2],[8,2],[8,3],[8,3],[8,3],[8,3],[8,3],[8,3],[8,3],[8,3],[8,4],[8,4],[8,4],[8,4],[8,4],[8,4],[8,4],[8,4],[8,5],[8,5],[8,5],[8,5],[8,5],[8,5],[8,5],[8,5],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[8,10],[9,8],[9,8],[9,8],[9,8],[9,8],[9,8],[9,8],[9,8],[9,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,3],[9,3],[9,3],[9,3],[9,3],[9,3],[9,3],[9,3],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,5],[9,5],[9,5],[9,5],[9,5],[9,5],[9,5],[9,5],[9,6],[9,6],[9,6],[9,6],[9,6],[9,6],[9,6],[9,6],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,-1],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[9,10],[10,9],[10,9],[10,9],[10,9],[10,9],[10,9],[10,9],[10,9],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,3],[10,3],[10,3],[10,3],[10,3],[10,3],[10,3],[10,3],[10,4],[10,4],[10,4],[10,4],[10,4],[10,4],[10,4],[10,4],[10,5],[10,5],[10,5],[10,5],[10,5],[10,5],[10,5],[10,5],[10,6],[10,6],[10,6],[10,6],[10,6],[10,6],[10,6],[10,6],[10,7],[10,7],[10,7],[10,7],[10,7],[10,7],[10,7],[10,7],[10,-1],[10,-1],[10,-1],[10,-1],[10,-1],[10,-1],[10,-1],[10,-1],[10,-1],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8]],\"vBucketMapForward\":[[0,5],[0,6],[0,7],[0,8],[0,9],[0,10],[0,10],[0,10],[0,10],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,8],[0,8],[0,8],[0,8],[0,8],[0,8],[0,8],[0,8],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[0,9],[2,0],[2,3],[2,4],[2,5],[2,7],[2,1],[2,8],[0,2],[0,3],[3,0],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[4,2],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[5,0],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[6,0],[6,3],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[7,0],[7,2],[7,4],[7,5],[7,6],[7,6],[7,6],[7,6],[1,0],[1,2],[1,3],[1,4],[1,6],[1,7],[1,7],[1,7],[8,2],[8,3],[8,4],[8,5],[8,1],[8,1],[8,1],[8,1],[8,1],[9,0],[9,2],[9,3],[9,4],[9,5],[9,6],[9,8],[9,8],[10,0],[10,3],[10,4],[10,5],[10,6],[10,7],[10,8],[10,9],[10,9],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,7],[2,7],[2,7],[2,7],[2,7],[2,7],[2,7],[2,7],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,8],[2,8],[2,8],[2,8],[2,8],[2,8],[2,8],[2,8],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,9],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[2,10],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[0,10],[0,10],[0,10],[0,10],[0,10],[0,10],[3,2],[3,2],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,7],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,8],[3,8],[3,8],[3,8],[3,8],[3,8],[3,8],[3,8],[3,9],[3,9],[3,9],[3,9],[3,9],[3,9],[3,9],[3,9],[3,10],[3,10],[3,10],[3,10],[3,10],[3,10],[3,10],[3,10],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[3,1],[3,8],[3,9],[3,10],[4,3],[4,3],[4,5],[4,7],[4,1],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,7],[4,7],[4,7],[4,7],[4,7],[4,7],[4,7],[4,7],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,8],[4,8],[4,8],[4,8],[4,8],[4,8],[4,8],[4,8],[4,9],[4,9],[4,9],[4,9],[4,9],[4,9],[4,9],[4,9],[4,10],[4,10],[4,10],[4,10],[4,10],[4,10],[4,10],[4,10],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[4,8],[4,9],[4,10],[5,4],[5,4],[5,6],[5,7],[5,1],[5,8],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,7],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,8],[5,8],[5,8],[5,8],[5,8],[5,8],[5,8],[5,8],[5,9],[5,9],[5,9],[5,9],[5,9],[5,9],[5,9],[5,9],[5,10],[5,10],[5,10],[5,10],[5,10],[5,10],[5,10],[5,10],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[5,9],[5,10],[6,5],[6,5],[6,5],[6,5],[6,7],[6,1],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,7],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,8],[6,9],[6,9],[6,9],[6,9],[6,9],[6,9],[6,9],[6,9],[6,10],[6,10],[6,10],[6,10],[6,10],[6,10],[6,10],[6,10],[7,0],[7,0],[7,0],[7,0],[7,0],[7,0],[7,0],[7,0],[6,9],[6,10],[7,6],[7,6],[7,6],[7,6],[7,6],[7,6],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,3],[7,4],[7,4],[7,4],[7,4],[7,4],[7,4],[7,4],[7,4],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,5],[7,1],[7,1],[7,1],[7,1],[7,1],[7,1],[7,1],[7,1],[7,1],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,8],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,9],[7,10],[7,10],[7,10],[7,10],[7,10],[7,10],[7,10],[7,10],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[7,1],[7,10],[1,7],[1,7],[1,7],[1,7],[1,7],[1,7],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,8],[1,8],[1,8],[1,8],[1,8],[1,8],[1,8],[1,8],[1,8],[1,9],[1,9],[1,9],[1,9],[1,9],[1,9],[1,9],[1,9],[1,9],[1,10],[1,10],[1,10],[1,10],[1,10],[1,10],[1,10],[1,10],[1,10],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[8,0],[1,7],[1,8],[8,1],[8,1],[8,1],[8,1],[8,1],[8,9],[8,2],[8,2],[8,2],[8,2],[8,2],[8,2],[8,2],[8,2],[8,3],[8,3],[8,3],[8,3],[8,3],[8,3],[8,3],[8,3],[8,4],[8,4],[8,4],[8,4],[8,4],[8,4],[8,4],[8,4],[8,5],[8,5],[8,5],[8,5],[8,5],[8,5],[8,5],[8,5],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,6],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,7],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,9],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[8,10],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[9,0],[8,10],[9,8],[9,8],[9,8],[9,8],[9,8],[9,8],[9,8],[9,8],[9,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,3],[9,3],[9,3],[9,3],[9,3],[9,3],[9,3],[9,3],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,5],[9,5],[9,5],[9,5],[9,5],[9,5],[9,5],[9,5],[9,6],[9,6],[9,6],[9,6],[9,6],[9,6],[9,6],[9,6],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,7],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[9,10],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[10,0],[9,10],[10,9],[10,9],[10,9],[10,9],[10,9],[10,9],[10,9],[10,9],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,2],[10,3],[10,3],[10,3],[10,3],[10,3],[10,3],[10,3],[10,3],[10,4],[10,4],[10,4],[10,4],[10,4],[10,4],[10,4],[10,4],[10,5],[10,5],[10,5],[10,5],[10,5],[10,5],[10,5],[10,5],[10,6],[10,6],[10,6],[10,6],[10,6],[10,6],[10,6],[10,6],[10,7],[10,7],[10,7],[10,7],[10,7],[10,7],[10,7],[10,7],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8],[10,8]]},\"nodes\":[{\"couchApiBase\":\"http://172.23.104.5:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.104.5:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.104.69:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.104.69:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.108.103:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.108.103:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.96.148:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.96.148:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.97.119:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.97.119:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.97.121:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.97.121:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.97.122:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.97.122:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.97.239:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.97.239:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.99.20:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.99.20:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.99.21:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.99.21:8091\",\"ports\":{\"direct\":11210}},{\"couchApiBase\":\"http://172.23.99.25:8092/NEW_ORDER%2Bbb98642244743421670c71047afa257a\",\"hostname\":\"172.23.99.25:8091\",\"ports\":{\"direct\":11210}}],\"nodesExt\":[{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.104.5\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.104.69\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.108.103\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.96.148\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.97.119\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.97.121\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"thisNode\":true,\"hostname\":\"172.23.97.122\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.97.239\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.99.20\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.99.21\"},{\"services\":{\"capi\":8092,\"capiSSL\":18092,\"kv\":11210,\"kvSSL\":11207,\"mgmt\":8091,\"mgmtSSL\":18091,\"projector\":9999},\"hostname\":\"172.23.99.25\"},{\"services\":{\"eventingAdminPort\":8096,\"eventingDebug\":9140,\"eventingSSL\":18096,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.104.137\"},{\"services\":{\"fts\":8094,\"ftsGRPC\":9130,\"ftsGRPCSSL\":19130,\"ftsSSL\":18094,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.104.155\"},{\"services\":{\"fts\":8094,\"ftsGRPC\":9130,\"ftsGRPCSSL\":19130,\"ftsSSL\":18094,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.104.157\"},{\"services\":{\"indexAdmin\":9100,\"indexHttp\":9102,\"indexHttps\":19102,\"indexScan\":9101,\"indexStreamCatchup\":9104,\"indexStreamInit\":9103,\"indexStreamMaint\":9105,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.104.67\"},{\"services\":{\"indexAdmin\":9100,\"indexHttp\":9102,\"indexHttps\":19102,\"indexScan\":9101,\"indexStreamCatchup\":9104,\"indexStreamInit\":9103,\"indexStreamMaint\":9105,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.104.70\"},{\"services\":{\"cbas\":8095,\"cbasSSL\":18095,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.106.188\"},{\"services\":{\"mgmt\":8091,\"mgmtSSL\":18091,\"n1ql\":8093,\"n1qlSSL\":18093},\"hostname\":\"172.23.120.245\"},{\"services\":{\"indexAdmin\":9100,\"indexHttp\":9102,\"indexHttps\":19102,\"indexScan\":9101,\"indexStreamCatchup\":9104,\"indexStreamInit\":9103,\"indexStreamMaint\":9105,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.121.117\"},{\"services\":{\"cbas\":8095,\"cbasSSL\":18095,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.121.3\"},{\"services\":{\"eventingAdminPort\":8096,\"eventingDebug\":9140,\"eventingSSL\":18096,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.123.27\"},{\"services\":{\"backupAPI\":8097,\"backupAPIHTTPS\":18097,\"backupGRPC\":9124,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.123.28\"},{\"services\":{\"mgmt\":8091,\"mgmtSSL\":18091,\"n1ql\":8093,\"n1qlSSL\":18093},\"hostname\":\"172.23.96.251\"},{\"services\":{\"indexAdmin\":9100,\"indexHttp\":9102,\"indexHttps\":19102,\"indexScan\":9101,\"indexStreamCatchup\":9104,\"indexStreamInit\":9103,\"indexStreamMaint\":9105,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.96.252\"},{\"services\":{\"indexAdmin\":9100,\"indexHttp\":9102,\"indexHttps\":19102,\"indexScan\":9101,\"indexStreamCatchup\":9104,\"indexStreamInit\":9103,\"indexStreamMaint\":9105,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.96.253\"},{\"services\":{\"cbas\":8095,\"cbasSSL\":18095,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.97.242\"},{\"services\":{\"eventingAdminPort\":8096,\"eventingDebug\":9140,\"eventingSSL\":18096,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.98.135\"},{\"services\":{\"indexAdmin\":9100,\"indexHttp\":9102,\"indexHttps\":19102,\"indexScan\":9101,\"indexStreamCatchup\":9104,\"indexStreamInit\":9103,\"indexStreamMaint\":9105,\"mgmt\":8091,\"mgmtSSL\":18091},\"hostname\":\"172.23.99.11\"}],\"clusterCapabilitiesVer\":[1,0],\"clusterCapabilities\":{\"n1ql\":[\"enhancedPreparedStatements\"]}}",
                              "message": "Some keys were not deleted [9C0118FE-81_1134092]",
                              "caller": "couchbase:1986",
                              "key": "datastore.couchbase.delete_failed"
                          }
                      ],
                      "scanConsistency": "unbounded",
                      "requestTime": "2021-09-29T13:57:54.27-07:00",
                      "statementType": "DELETE",
                      "phaseOperators": {
                          "authorize": 1,
                          "delete": 1
                      },
                      "serviceTime": "3.095512524s",
                      "remoteAddr": "172.23.98.135:59264",
                      "elapsedTime": "3.095606797s",
                      "resultSize": 0,
                      "n1qlFeatCtrl": 76,
                      "statement": "DELETE from NEW_ORDER.event_0.`coll1` USE KEYS $meta.id ;",
                      "requestId": "e38f2b85-3234-4aeb-b5cc-8d7f40b1a198",
                      "clientContextID": "14@n1ql_0.js(OnDelete)",
                      "userAgent": "libcouchbase/3.1.4 (Linux-4.15.0-156-generic; x86_64; GNU 10.2.0)",
                      "useCBO": true,
                      "resultCount": 0,
                      "errorCount": 1,
                      "phaseTimes": {
                          "authorize": "24.254\u00b5s",
                          "run": "3.094894421s",
                          "instantiate": "29.416\u00b5s",
                          "parse": "518.139\u00b5s",
                          "plan": "44.845\u00b5s",
                          "delete": "3.094721137s"
                      }
                  }
              }
          ],
          "requestID": "9a788cf9-6171-46d8-9cb6-fc592c07508f",
          "signature": {
              "*": "*"
          }

      The dump from completed_requests is attached.

      Query nodes : 172.23.96.251, 172.23.120.245
      Eventing nodes : 172.23.104.137, 172.23.123.27, 172.23.98.135

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty