Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39294

[CX] Verify all files are retrieved from S3 bucket when processing external dataset

    XMLWordPrintable

Details

    Description

      As per the AWS documentation,

      https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html#listObjects-software.amazon.awssdk.services.s3.model.ListObjectsRequest-

      there is a limit of 1000 objects when listing objects from an S3 bucket. We should verify that the code that lists the objects retrieves all the objects in the S3 bucket. We should do a one-off test on S3, and QE should have an automated test for such a scenario where a bucket has more than 1000 files.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          A test for this ticket has been passed to QE team

          Hussain.Towaileb Hussain Towaileb added a comment - A test for this ticket has been passed to QE team
          Hussain.Towaileb Hussain Towaileb added a comment - Commit reference for the code change: https://github.com/couchbase/asterixdb/commit/1a28c658810ded5d50ff88f42ef88ec8f82926e1

          Build couchbase-server-6.6.0-7643 contains cbas-core commit 3360e0f with commit message:
          MB-39294: Ensure retrieving all objects if an S3 bucket has more than 1000 objects

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7643 contains cbas-core commit 3360e0f with commit message: MB-39294 : Ensure retrieving all objects if an S3 bucket has more than 1000 objects

          Build couchbase-server-7.0.0-2158 contains cbas-core commit 3360e0f with commit message:
          MB-39294: Ensure retrieving all objects if an S3 bucket has more than 1000 objects

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2158 contains cbas-core commit 3360e0f with commit message: MB-39294 : Ensure retrieving all objects if an S3 bucket has more than 1000 objects
          umang.agrawal Umang added a comment -

          verified cb server version 6.6.0-4861

          umang.agrawal Umang added a comment - verified cb server version 6.6.0-4861

          Build couchbase-server-6.6.2-9599 contains cbas-core commit 3360e0f with commit message:
          MB-39294: Ensure retrieving all objects if an S3 bucket has more than 1000 objects

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 3360e0f with commit message: MB-39294 : Ensure retrieving all objects if an S3 bucket has more than 1000 objects

          People

            umang.agrawal Umang
            ali.alsuliman Ali Alsuliman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty