Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42783

Azure upload/download pool needs to be using multipart block uploads/downloads

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • Critical
    • Resolution: Done
    • Cheshire-Cat
    • 7.0.0
    • tools
    • None

    Description

      What's the issue?
      The initial implementation of the Azure upload pool used the 'PutBlob' API call by design so that we could get things implemented, however, it's currently limited to uploading 256 MiB of data, in the long run we should upgrade to using a multipart uploading implementation for larger than normal blocks (in a similar fashion to how we do with S3). See the remarks section of the documentation for more information.

      What's the fix?
      Upload larger files using multipart uploads.

      Update this size seems to have been changed in the latest version of the SDK (which we're not running yet); either way, the new limit is still not high enough for our needs, so we still need to use multipart uploads.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            james.lee James Lee added a comment -

            Added link to the documentation; it looks like generic uploading of files (as block blobs) is already implemented by the SDK (similar to the Uploader from the aws-sdk-go). We should look at using this implementation if it will suit our needs over reinventing the wheel.

            james.lee James Lee added a comment - Added link to the documentation; it looks like generic uploading of files (as block blobs) is already implemented by the SDK (similar to the Uploader from the aws-sdk-go). We should look at using this implementation if it will suit our needs over reinventing the wheel.

            Build couchbase-server-7.0.0-3994 contains backup commit 71cfb60 with commit message:
            MB-42783 Use multipart uploads/downloads for upload/download pools

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3994 contains backup commit 71cfb60 with commit message: MB-42783 Use multipart uploads/downloads for upload/download pools

            People

              james.lee James Lee
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty