Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-731

cbc-pillowfight: smoother batching

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 3.0.0-beta1
    • 2.6.4
    • None
    • None

    Description

      pillowfight is a marvellous tool for stressing clusters. However it can be hard to get a smooth rate-limit, particularly as the batch size increases. With a batch size of 1000, I need to wait for all 1000 ops to return before dispatching the next batch. This results in rather 'bursty' behaviour at the Server.

      As part of the Google Performance benchmark we ran, we invested some time developing a token-style system where ops were allowed to be dispatched, before the previous batch had completed. See https://github.com/dhaikney/libcouchbase/commits/durability_tokens
      This allowed more ops in flight at any one time and ultimately smoother performance and easier benchmarking.

      Raising this ticket to discuss possible ways of incorporating this kind of behaviour into master. Perhaps we could pick a threshold of batch size (20%) and when 20% of the batch has returned, issue those 20% of ops again (subject to rate-limit). Maybe this threshold could be a parameter (though fewer CLI options the better)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              avsej Sergey Avseyev
              dhaikney David Haikney (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty