Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
7.6.0
-
Enterprise Edition 7.6.0 build 1763
-
Untriaged
-
Linux x86_64
-
0
-
Yes
Description
Script to Repro
./sequoia -client 172.23.104.27:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.6.0-1763 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
|
Almost within an hour of starting the run we saw crashes like the following repeatedly.
ns_1@172.23.96.203 6:36:57 AM 7 Nov, 2023
Service 'backup' exited with status 1. Restarting. Messages:
|
2023-11-07T06:36:49.524-08:00 INFO (Main) Running node version backup-7.6.0-1763-325c2db with options: -http-port=8097 -grpc-port=9124 -https-port=18097 -cert-path=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem -key-path=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem -ca-path=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem -client-cert-path=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem -client-key-path=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=off -cbm=/opt/couchbase/bin/cbbackupmgr -node-uuid=d9339f7f9b3061ef2a958a5806122dcc -public-address=172.23.96.203 -admin-port=8091 -log-file=none -log-level=debug -integrated-mode -integrated-mode-host=http://127.0.0.1:8091 -secure-integrated-mode-host=https://127.0.0.1:18091 -integrated-mode-user=@backup -tmp-dir=/opt/couchbase/var/lib/couchbase/tmp -cbauth-host=127.0.0.1:8091
|
2023-11-07T06:36:49.524-08:00 INFO (Main) Initialized logger {"log level": "debug"}
|
2023-11-07T06:36:49.527-08:00 DEBUG (Main) File limit info {"curr": 200000, "max": 200000, "err": null}
|
2023-11-07T06:36:52.028-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T06:36:54.529-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T06:36:57.030-08:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: failed to set auth headers: exhausted retry count after 3 attempts: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T06:36:57.031-08:00 ERROR (Main) Failed to run node {"err": "could not create REST client: failed to bootstrap client: failed to connect to any host(s) from the connection string, check username and password"}
|
172.23.96.203: backup
2023-11-07T05:52:44.540-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T05:52:47.041-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T05:52:49.542-08:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: failed to set auth headers: exhausted retry count after 3 attempts: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T05:52:52.325-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T05:52:54.827-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
|
2023-11-07T05:52:57.327-08:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: failed to set auth headers: exhausted retry count after 3 attempts: Unable to find given hostport in cbauth database: `localhost:8091'
|
cbcollect_info attached. I don't think we saw it on the run we had on 7.6.0-1756
Attachments
Issue Links
- is caused by
-
MB-59360 [CBBS][System Test] Starting a backup can fail due to stale authentication
- Closed