Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49671

Backup Service: Unknown error message displayed during Create Repository with GCP if cloud storage bucket gets 403

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 7.1.0
    • tools
    • None
    • Triaged
    • 1
    • Unknown

    Description

      Backup Service with Couchbase server (Enterprise Edition 7.1.0 build 1695) running in a GCP VM.

      During Create Repository, using Cloud Auth Type=Instance metadata service – if a name of a GCP cloud bucket that is not in the same project as the VM that the Backup Service is running on is given, you see a "Could not add repository: Unknown error ocurred please refresh page or try again later."   You also see a stack in the /opt/couchbase/var/lib/couchbase/logs/backup_service.log .

      What was expected was an error message conveying similar info as cbbackupmgr CLI when you do the same thing on the same VM command line.

      Example:

      $ cbbackupmgr config -a gs://cb_hjv_g_bucket1 -r gcp_vm_test_repo --obj-staging-dir /home/hyun_ju_vega/couchbase_staging --obj-auth-by-instance-metadata
      Backup repository creation failed: failed to lock remote archive: failed to create lockfile object with key 'lock-fad135ce-b8e6-40f1-8965-ca1e7e72d558.
      lk': googleapi: Error 403: 350900750131-compute@developer.gserviceaccount.com does not have storage.objects.create access to the Google Cloud Storage object., forbidden

      Attaching the backup_service.log that has a stack that looks like this:

      021-11-19T05:23:51.227Z INFO (REST) POST /api/v1/cluster/self/repository/active/repo-hjv-test-project-vm1
      2021-11-19T05:23:51.284Z ERROR (Janitor) Location check failed to create node file {"err": "could not create node file due to a cloud error: googleapi: Error 403: 350900750131-compute@developer.gserviceaccount.com does not have storage.objects.create access to the Google Cloud Storage object., forbidden"}
      2021/11/19 05:23:51 http: panic serving 127.0.0.1:58564: path must not end with "/"
      goroutine 13343 [running]:
      net/http.(*conn).serve.func1()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17/go/src/net/http/server.go:1801 +0xb9
      panic({0xe715e0, 0x1233cf0})
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17/go/src/runtime/panic.go:1047 +0x266
      github.com/couchbase/cbauth/metakv.assertValidPath(...)
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/cbauth@v0.0.0-20210911011937-6a860d4b3951/metakv/metakv.go:159
      github.com/couchbase/cbauth/metakv.(*store).delete(0xdc3918, {0x104a519, 0x18}, {0x0, 0x0})
      ...
      ...
      ...
      net/http.(*conn).serve(0xc0000d20a0, {0x1264590, 0xc0003a8b10})
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17/go/src/net/http/server.go:1929 +0xb08
      created by net/http.(*Server).Serve
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17/go/src/net/http/server.go:3033 +0x4e8

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              hyun-ju.vega Hyun-Ju Vega
              hyun-ju.vega Hyun-Ju Vega
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty