Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
6.6.2->9588 to 7.0.0-5033
-
Untriaged
-
Centos 64-bit
-
1
-
No
Description
Steps to Repro
1. Run longevity system test on 6.6.2 for 3 days. It initially had 27 node cluster.
./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. Swap rebalance 6 nodes one of each service type.
3. Now failover 6 nodes, one of each service type.
4. sysemctl stop couchbas-server on 6 nodes.
5. Install 7.0.0 on all of them.
6. Do a recovery + rebalance
7. Now failover 4 nodes, one of each service type.(172.23.106.117 172.23.105.25 172.23.105.210 172.23.105.206)
8. Do a delta recovery(for kv) + rebalance.
The above steps were repeated few times and finally we did upgrade using regular rebalance, i.e rebalance in of all the 7.0 nodes followed by another rebalance where we remove all the 6.6.2 nodes.
Once the entire cluster got upgrade saw error messages like the following on the fts nodes.
Steps to Repro
1. Run longevity system test on 6.6.2 for 3 days. It initially had 27 node cluster.
./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. Swap rebalance 6 nodes one of each service type.
3. Now failover 6 nodes, one of each service type.
4. sysemctl stop couchbas-server on 6 nodes.
5. Install 7.0.0 on all of them.
6. Do a recovery + rebalance
7. Now failover 4 nodes, one of each service type.(172.23.106.117 172.23.105.25 172.23.105.210 172.23.105.206)
8. Do a delta recovery(for kv) + rebalance.
The above steps were repeated few times and finally we did upgrade using regular rebalance, i.e rebalance in of all the 7.0 nodes followed by another rebalance where we remove all the 6.6.2 nodes.
Once the entire cluster got upgrade saw error messages like the following on the eventing nodes.
172.23.105.25 : eventing
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:28.891-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:84451] { "message": " Query error : {\n\"requestID\": \"93d9f004-4dba-4bcc-92f5-3f85479365a2\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"12.707274ms\",\"executionTime\": \"12.589882ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
172.23.105.29 : eventing
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:24.948-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"02bae73a-86d9-403d-b9c7-99046cf75db3\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"9.492316ms\",\"executionTime\": \"9.422208ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:24.969-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"c0296cce-372e-4426-b659-b1a45db7788e\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"6.101539ms\",\"executionTime\": \"6.039171ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:24.987-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"b6302478-3e94-4bc0-ac0d-2c1243001f94\",\n\"clientContextID\": \"34@timer_op.js(timerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"5.289011ms\",\"executionTime\": \"5.225222ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.timerCallback (timer_op.js:34:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.timerCallback (timer_op.js:34:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.133-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"75545e13-02e0-419f-93de-f6c2d0a5d8cc\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"7.722293ms\",\"executionTime\": \"7.658904ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.206-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"6d0cb63c-1aa7-4588-8e5d-1d2abf81b732\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"11.168083ms\",\"executionTime\": \"11.079673ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.382-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"4dd3504a-43a3-477b-8ddf-461c049cea84\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"5.04405ms\",\"executionTime\": \"4.984229ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:25.420-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"a2441c8b-4840-4117-afd3-3537b0789ee1\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"12.426987ms\",\"executionTime\": \"12.365891ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:29.396-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"2e18b189-759a-46e3-9224-9a17ed6fe94d\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"2.574803ms\",\"executionTime\": \"2.518376ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:54:29.407-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:116019] { "message": " Query error : {\n\"requestID\": \"2f5bb863-f04e-4519-9665-c122b0e77670\",\n\"clientContextID\": \"34@timer_op.js(timerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"3.349757ms\",\"executionTime\": \"3.305541ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at Function.timerCallback (timer_op.js:34:24)"} Line: 33 Code: var select_query = N1QL('SELECT * FROM default USE KEYS[$context.docID];', {'$context':context}, { 'consistency': 'none' }); Stack: Error at Function.timerCallback (time
|
r_op.js:34:24)
|
172.23.106.246 : eventing
/opt/couchbase/var/lib/couchbase/logs/eventing.log:2021-04-27T06:50:36.947-07:00 [Info] eventing-consumer [worker_timer_op_0:/tmp/127.0.0.1:8091_0_272693763.sock:74764] { "message": " Query error : {\n\"requestID\": \"72d0d747-2256-4548-985a-b3cabf813080\",\n\"clientContextID\": \"25@timer_op.js(NDtimerCallback)\",\n\"signature\": {\"*\":\"*\"},\n\"results\": [\n],\n\"errors\": [{\"code\":13014,\"msg\":\"User does not have credentials to run SELECT queries on default:default. Add role query_select on default:default to allow the query to run.\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"3.219975ms\",\"executionTime\": \"3.154695ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 3,\"errorCount\": 1}\n}\n", "stack": "Error\n at N1QL (<anonymous>)\n at Function.NDtimerCallback (timer_op.js:25:24)"} Line: 4294967295 Stack: Error at N1QL (<anonymous>) at Function.NDtimerCallback (timer_op.js:25:24)
|
cbcollect_info attached.