Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38316

Connection failure when creating over 500 collections on a restore

    XMLWordPrintable

Details

    Description

      What's the issue?
      When creating a large number of collections during a restore (for me, over 500 collections) it's highly likely that you will eventually get a connection failure (which isn't explicitly handled in our retry logic meaning we exit early).

      What's the solution?
      There are multiple possible solutions to the problem:
      1) We explicitly handle the connection failure and retry to create the scope/collection
      2) We wait until there is a bulk creation endpoint which we can use to create the collections using a single request

      The preferred solution here is the second, not only being it solves this problem but because it will also alleviate the severe performance issue I encountered whilst attempting to reproduce this issue. Below are some basic benchmarks which display how long is takes to create 'n' collections.
      1) Creating 10 collections takes 6.03 seconds
      2) Creating 100 collections takes 90.04 seconds
      3) Creating 500 collections takes 506.10 seconds
      4) Creating more than 500 generally results in a connection failure.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              james.lee James Lee
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty