Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61417

Add flag for COPY TO S3 to skip write permission validation

    XMLWordPrintable

Details

    • 0
    • Analytics Sprint 40, Analytics Sprint 41

    Description

      Currently, for COPY TO S3, one of our early checks validates that:

      • We have the write permission, by writing a small test file to the destination, then deleting it, which also tests that
      • Multipart upload is supported, as this is the form of operation we use for writing.

       

      This test helps identify failure scenarios early before processing (for example) large amount of data, only to fail eventually.

      The limitation of this validation is that it is not always possible to know the final destination we will be writing to, depending on the statement itself. And so, we always perform this test at the root of the bucket.

      According to the S3 docs, it seems it might be possible to grant the permissions only on specific prefixes in a bucket, but not all. This can be problematic if we have permission on our destination prefix, but not the root of the bucket, because the validation will keep trying to validate against the root, which will lead to a failure of the operation.

      This ticket suggests the addition of a flag, e.g. "validate-write-permission" (default to true), to skip this operation and let the user carry the responsibility of having the right permissions at the right time. Note this will only skip the write permission validation, all other validations will still take place.

      Attachments

        Issue Links

          Activity

            People

              Hussain.Towaileb Hussain Towaileb
              Hussain.Towaileb Hussain Towaileb
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty