Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45947

[System test upgrade] : Eventing failing with "User does not have credentials to run SELECT queries on default:default.User does not have credentials to run SELECT queries on default:default." post upgrade to 7.0.0

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Steps to Repro
      1. Run longevity system test on 6.6.2 for 3 days. It initially had 27 node cluster.

       ./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

      2. Swap rebalance 6 nodes one of each service type.
      3. Now failover 6 nodes, one of each service type.
      4. sysemctl stop couchbas-server on 6 nodes.
      5. Install 7.0.0 on all of them.
      6. Do a recovery + rebalance
      7. Now failover 4 nodes, one of each service type.(172.23.106.117 172.23.105.25 172.23.105.210 172.23.105.206)
      8. Do a delta recovery(for kv) + rebalance.
      The above steps were repeated few times and finally we did upgrade using regular rebalance, i.e rebalance in of all the 7.0 nodes followed by another rebalance where we remove all the 6.6.2 nodes.

      Once the entire cluster got upgrade saw error messages like the following on the fts nodes.
      Steps to Repro
      1. Run longevity system test on 6.6.2 for 3 days. It initially had 27 node cluster.

       ./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

      2. Swap rebalance 6 nodes one of each service type.
      3. Now failover 6 nodes, one of each service type.
      4. sysemctl stop couchbas-server on 6 nodes.
      5. Install 7.0.0 on all of them.
      6. Do a recovery + rebalance
      7. Now failover 4 nodes, one of each service type.(172.23.106.117 172.23.105.25 172.23.105.210 172.23.105.206)
      8. Do a delta recovery(for kv) + rebalance.
      The above steps were repeated few times and finally we did upgrade using regular rebalance, i.e rebalance in of all the 7.0 nodes followed by another rebalance where we remove all the 6.6.2 nodes.

      Once the entire cluster got upgrade saw error messages like the following on the eventing nodes.

      172.23.105.25 : eventing

      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:28.891-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:84451] {  "message": " Query error : {\n\"requestID\": \"93d9f004-4dba-4bcc-92f5-3f85479365a2\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"12.707274ms\",\"executionTime\": \"12.589882ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)

      172.23.105.29 : eventing

      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:24.948-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"02bae73a-86d9-403d-b9c7-99046cf75db3\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"9.492316ms\",\"executionTime\": \"9.422208ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:24.969-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"c0296cce-372e-4426-b659-b1a45db7788e\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"6.101539ms\",\"executionTime\": \"6.039171ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:24.987-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"b6302478-3e94-4bc0-ac0d-2c1243001f94\",\n\"clientContextID\": \"34@timer_op.js(timerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"5.289011ms\",\"executionTime\": \"5.225222ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.timerCallback (timer_op.js:34:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.timerCallback (timer_op.js:34:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.133-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"75545e13-02e0-419f-93de-f6c2d0a5d8cc\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"7.722293ms\",\"executionTime\": \"7.658904ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.206-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"6d0cb63c-1aa7-4588-8e5d-1d2abf81b732\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"11.168083ms\",\"executionTime\": \"11.079673ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.382-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"4dd3504a-43a3-477b-8ddf-461c049cea84\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"5.04405ms\",\"executionTime\": \"4.984229ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.420-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"a2441c8b-4840-4117-afd3-3537b0789ee1\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"12.426987ms\",\"executionTime\": \"12.365891ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:29.396-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"2e18b189-759a-46e3-9224-9a17ed6fe94d\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"2.574803ms\",\"executionTime\": \"2.518376ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)
      		 
      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:29.407-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] {  "message": " Query error : {\n\"requestID\": \"2f5bb863-f04e-4519-9665-c122b0e77670\",\n\"clientContextID\": \"34@timer_op.js(timerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"3.349757ms\",\"executionTime\": \"3.305541ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at Function.timerCallback (timer_op.js:34:24)"} Line: 33 Code: var select_query = N1QL('SELECT * FROM default USE KEYS[$context.docID];', {'$context':context}, { 'consistency': 'none' }); Stack: Error    at Function.timerCallback (time
      		 
      r_op.js:34:24)

      172.23.106.246 : eventing

      /opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:50:36.947-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:74764] {  "message": " Query error : {\n\"requestID\": \"72d0d747-2256-4548-985a-b3cabf813080\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"3.219975ms\",\"executionTime\": \"3.154695ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at N1QL (<anonymous>)\n    at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error    at N1QL (<anonymous>)    at Function.NDtimerCallback (timer_op.js:25:24)

      cbcollect_info attached.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ankit.prabhu Ankit Prabhu
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty