Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59494

[System Test] :- Backup service repeatedly crashing in longevity test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 7.6.0
    • 7.6.0
    • tools

    • Enterprise Edition 7.6.0 build 1763
    • Untriaged
    • Linux x86_64
    • 0
    • Yes

    Description

      Script to Repro

      ./sequoia -client 172.23.104.27:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.6.0-1763 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
      

      Almost within an hour of starting the run we saw crashes like the following repeatedly.

      ns_1@172.23.96.203 6:36:57 AM 7 Nov, 2023

      Service 'backup' exited with status 1. Restarting. Messages:
      2023-11-07T06:36:49.524-08:00 INFO (Main) Running node version backup-7.6.0-1763-325c2db with options: -http-port=8097 -grpc-port=9124 -https-port=18097 -cert-path=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem -key-path=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem -ca-path=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem -client-cert-path=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem -client-key-path=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=off -cbm=/opt/couchbase/bin/cbbackupmgr -node-uuid=d9339f7f9b3061ef2a958a5806122dcc -public-address=172.23.96.203 -admin-port=8091 -log-file=none -log-level=debug -integrated-mode -integrated-mode-host=http://127.0.0.1:8091 -secure-integrated-mode-host=https://127.0.0.1:18091 -integrated-mode-user=@backup -tmp-dir=/opt/couchbase/var/lib/couchbase/tmp -cbauth-host=127.0.0.1:8091
      2023-11-07T06:36:49.524-08:00 INFO (Main) Initialized logger {"log level": "debug"}
      2023-11-07T06:36:49.527-08:00 DEBUG (Main) File limit info {"curr": 200000, "max": 200000, "err": null}
      2023-11-07T06:36:52.028-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T06:36:54.529-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T06:36:57.030-08:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: failed to set auth headers: exhausted retry count after 3 attempts: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T06:36:57.031-08:00 ERROR (Main) Failed to run node {"err": "could not create REST client: failed to bootstrap client: failed to connect to any host(s) from the connection string, check username and password"}
      

      172.23.96.203: backup

      2023-11-07T05:52:44.540-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T05:52:47.041-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T05:52:49.542-08:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: failed to set auth headers: exhausted retry count after 3 attempts: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T05:52:52.325-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T05:52:54.827-08:00 WARN (REST) (Attempt 8831872) Failed to get credentials due to error: Unable to find given hostport in cbauth database: `localhost:8091'
      2023-11-07T05:52:57.327-08:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: failed to set auth headers: exhausted retry count after 3 attempts: Unable to find given hostport in cbauth database: `localhost:8091'
      

      cbcollect_info attached. I don't think we saw it on the run we had on 7.6.0-1756

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty