[System Test] Queries within an Eventing function are failing with LCB_ERR_HTTP (1053) Error authorizing against cluster cause: Failure to authenticate user error

./sequoia -client 172.23.104.27:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.6.0-1694 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true

Day - 3

Cycle - 2

Scale - 3

Cluster configuration -

######  backup : 1 ===== > [172.23.104.215:8091]  ###########

######  eventing : 2 ===== > [172.23.104.227:8091 172.23.97.243:8091]  ###########

######  n1ql : 2 ===== > [172.23.105.237:8091 172.23.106.164:8091]  ###########

######  cbas : 3 ===== > [172.23.105.238:8091 172.23.121.87:8091 172.23.97.242:8091]  ###########

######  kv : 10 ===== > [172.23.106.108:8091 172.23.106.110:8091 172.23.121.193:8091 172.23.124.27:8091 172.23.96.186:8091 172.23.96.251:8091 172.23.97.189:8091 172.23.97.229:8091 172.23.97.244:8091 172.23.97.245:8091]  ###########

######  fts : 3 ===== > [172.23.106.124:8091 172.23.121.94:8091 172.23.96.170:8091]  ###########

######  index : 6 ===== > [172.23.104.213:8091 172.23.106.121:8091 172.23.120.59:8091 172.23.121.72:8091 172.23.96.252:8091 172.23.96.253:8091]  ###########

Error

2023-10-27T22:31:54.936-07:00 [INFO] "Query failed: " {"message":"SDK error : LCB_ERR_HTTP (1053) Query error : {\n\"requestID\": \"eb5dec75-f666-478c-8b56-de7edc919c8e\",\n\"clientContextID\": \"45@n1ql_0.js(OnUpdate)\",\n\"errors\": [{\"code\":2120,\"msg\":\"Error authorizing against cluster cause: Failure to authenticate user\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"965.289µs\",\"executionTime\": \"56.687µs\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 0,\"errorCount\": 1}\n}\n","stack":"Error\n    at OnUpdate (n1ql_0.js:45:23)"}

Note

Marking as regression since we did not see this issue on 7.2.x branch as well as on earlier Trinity builds. Test has been updated to incorporate 7.6 features but mostly likely this issue is unrelated to those script changes.

—

Steps to Reproduce the Issue (from Rishit Chaudhary)

Cluster Setup

1. Create a three Node cluster with the following service to node mapping
   1. Node 0: Runs the data service (Port: 9000)
   2. Node 1: Runs the eventing service (Port: 9001)
   3. Node 2: Runs the query service (Port: 9002)

Steps

Note: Jira appears to be causing issues with the numbering of the below steps. If the numbers appear incorrect, just follow the steps from top to bottom. The indented points are connected to the parent unindented point.

1. Create a new bucket called newBucket
2. Create an eventing function called newFunc defined as follows:

   1. function OnUpdate(doc, meta) {

      var a = SELECT 1;

      log("Doc created/updated", meta.id, a);

      }

      function OnDelete(meta, options) {

      var a = SELECT 1;

      log("Doc deleted/expired", meta.id, a);

      }

1. 1. 1. Note: The eventing function definition aims to ensure we make a call to the query service when we receive a mutation.
   2. Create/Modify a document to trigger running the eventing's OnUpdate function
      1. This operation should run successfully
   3. Rotate the credentials of the node running the query service (Port: 9002)

      1. curl -v -u Administrator 'http://127.0.0.1:9002/node/controller/rotateInternalCredentials' -X POST

1. 1. Create/Modify a document to trigger running the eventing's OnUpdate function
      1. This operation will fail with the failure count incremented by 2
      2. The failure message will be as follows,

         1. 2023-12-01T13:31:33.615+05:30 [INFO] {"count":1,"exception":"{\n \"message\": \"SDK error : LCB_ERR_HTTP (1053) Query error : {\\n\\\"requestID\\\": \\\"4ef2d231-4aa8-4c00-a48d-fcbf034f927c\\\",\\n\\\"clientContextID\\\": \\\"3@newFunc.js(OnUpdate)\\\",\\n\\\"errors\\\": [{\\\"code\\\":2120,\\\"msg\\\":\\\"Error authorizing against cluster cause: Failure to authenticate user\\\"}],\\n\\\"status\\\": \\\"fatal\\\",\\n\\\"metrics\\\": {\\\"elapsedTime\\\": \\\"688.333µs\\\",\\\"executionTime\\\": \\\"39.625µs\\\",\\\"resultCount\\\": 0,\\\"resultSize\\\": 0,\\\"serviceLoad\\\": 0,\\\"errorCount\\\": 1}\\n}\\n\",\n \"stack\": \"Error\\n at OnUpdate (newFunc.js:3:23)\"\n}","file":"newFunc.js","line":2,"since":"2023-12-01T13:31:28","srcLine":"var a = couchbase.n1qlQuery('SELECT 1;', {}, { 'consistency': 'none' });","stack":"Error\n at OnUpdate (newFunc.js:3:23)"}