Details
-
Task
-
Resolution: Fixed
-
Major
-
Columnar 1.0.0
-
0
-
Analytics Sprint 40, Analytics Sprint 41
Description
Currently, for COPY TO S3, one of our early checks validates that:
- We have the write permission, by writing a small test file to the destination, then deleting it, which also tests that
- Multipart upload is supported, as this is the form of operation we use for writing.
This test helps identify failure scenarios early before processing (for example) large amount of data, only to fail eventually.
The limitation of this validation is that it is not always possible to know the final destination we will be writing to, depending on the statement itself. And so, we always perform this test at the root of the bucket.
According to the S3 docs, it seems it might be possible to grant the permissions only on specific prefixes in a bucket, but not all. This can be problematic if we have permission on our destination prefix, but not the root of the bucket, because the validation will keep trying to validate against the root, which will lead to a failure of the operation.
This ticket suggests the addition of a flag, e.g. "validate-write-permission" (default to true), to skip this operation and let the user carry the responsibility of having the right permissions at the right time. Note this will only skip the write permission validation, all other validations will still take place.
Attachments
Issue Links
- links to