Uploaded image for project: 'Couchbase Go SDK'
  1. Couchbase Go SDK
  2. GOCBC-1500

Bootstrap is infinitely retried if previously succeeded and now select bucket fails

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Major
    • None
    • core-10.2.8, core-10.2.0
    • core-library
    • None
    • 0

    Description

      Whilst testing what happens when the bucket cbimport is importing into is deleted some way into the import (MB-58093) I noticed that we seem to be infinitely retrying bootstrapping:

      2023-10-13T13:49:18.227+00:00 WARN: (Pool) (travel-sample) Failed to send document with key '<ud>route_14627</ud>' because of an unexpected EOF -- couchbase.(*MemcachedWorker).processOperation() at pool_worker.go:274
      2023-10-13T13:49:20.062+00:00 (Gocbcore) Pipeline Client 0xc000a84a20 failed to bootstrap: authentication failure - possible reasons - incorrect authentication configuration, bucket doesn’t exist or bucket may be hibernated | {"status_code":36,"document_key":"travel-sample","bucket":"travel-sample","error_name":"EACCESS","error_description":"Not authorized for command","opaque":6,"last_dispatched_to":"localhost:12000","last_dispatched_from":"127.0.0.1:44834","last_connection_id":"16314dcf5df31c4d/025376ba995ab4c5"}
      2023-10-13T13:49:22.464+00:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. authentication failure - possible reasons - incorrect authentication configuration, bucket doesn’t exist or bucket may be hibernated | {"status_code":36,"document_key":"travel-sample","bucket":"travel-sample","error_name":"EACCESS","error_description":"Not authorized for command","opaque":6,"last_dispatched_to":"localhost:12000","last_dispatched_from":"127.0.0.1:44834","last_connection_id":"16314dcf5df31c4d/025376ba995ab4c5"}
      2023-10-13T13:49:22.464+00:00 (Gocbcore) CCCPPOLL: Failed to retrieve config from any node.
      2023-10-13T13:49:22.598+00:00 (Gocbcore) Pipeline Client 0xc000a84a20 failed to bootstrap: authentication failure - possible reasons - incorrect authentication configuration, bucket doesn’t exist or bucket may be hibernated | {"status_code":36,"document_key":"travel-sample","bucket":"travel-sample","error_name":"EACCESS","error_description":"Not authorized for command","opaque":6,"last_dispatched_to":"localhost:12000","last_dispatched_from":"127.0.0.1:44842","last_connection_id":"16314dcf5df31c4d/cf42fd96e2caed50"}
      2023-10-13T13:49:25.130+00:00 (Gocbcore) Pipeline Client 0xc000a84a20 failed to bootstrap: authentication failure - possible reasons - incorrect authentication configuration, bucket doesn’t exist or bucket may be hibernated | {"status_code":36,"document_key":"travel-sample","bucket":"travel-sample","error_name":"EACCESS","error_description":"Not authorized for command","opaque":6,"last_dispatched_to":"localhost:12000","last_dispatched_from":"127.0.0.1:44848","last_connection_id":"16314dcf5df31c4d/dfb9c4898677622d"}
      2023-10-13T13:49:27.657+00:00 (Gocbcore) Pipeline Client 0xc000a84a20 failed to bootstrap: authentication failure - possible reasons - incorrect authentication configuration, bucket doesn’t exist or bucket may be hibernated | {"status_code":36,"document_key":"travel-sample","bucket":"travel-sample","error_name":"EACCESS","error_description":"Not authorized for command","opaque":6,"last_dispatched_to":"localhost:12000","last_dispatched_from":"127.0.0.1:44864","last_connection_id":"16314dcf5df31c4d/2422a600d703b7e7"}
      2023-10-13T13:49:27.968+00:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. authentication failure - possible reasons - incorrect authentication configuration, bucket doesn’t exist or bucket may be hibernated | {"status_code":36,"document_key":"travel-sample","bucket":"travel-sample","error_name":"EACCESS","error_description":"Not authorized for command","opaque":6,"last_dispatched_to":"localhost:12000","last_dispatched_from":"127.0.0.1:44864","last_connection_id":"16314dcf5df31c4d/2422a600d703b7e7"}
      2023-10-13T13:49:27.968+00:00 (Gocbcore) CCCPPOLL: Failed to retrieve config from any node.
      

      Digging in a little I tested to see how retries are normally handled and it seems if you have an initial failure it is handled by the diagnostics component (an error is set here and whether to retry is checked here). In testing I found that if I had already successfully bootstrapped once then the error was not set, so I assume the diagnostics was not being run for some reason.

      Reproduction

      1. Create a cluster with one KV node with creds Administrator:asdasd
      2. Create a bucket named default
      3. Run attached reproduction program and notice it does not terminate despite timeouts being set

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Matt.Hall Matt Hall
              Matt.Hall Matt Hall
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty