Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7505

Synchronous writes seem to collapse the database

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 3.0
    • Component/s: clients
    • Security Level: Public
    • Labels:
    • Environment:
      4-node cluster, 32GB RAM per node, each machine with 4 SSDs.
    • Triage:
      Untriaged

      Description

      When performing updates against the database, we're able to get throughput of about 160,000 ops/second, with throughput to disk of about 21,000 ops/second (92% cache hit rate).

      However, when we try to do either synchronous writes to disk or synchronous replication, the server fails. Either we get 100% error rates, or if we throttle input and add auto-retries, the throughput drops to tens of operations per second. Given the disk is happily doing over 20k ops/sec, we'd expect to be able to process traffic at that volume even with synchronous disk writes.

      We're calling the update() method in the Java client with PersistTo.ONE (or ReplicateTo.ONE). This is done through YCSB.

      Is this the correct approach? Is there something else we should be doing? What numbers should we be expecting?

      Attached is the console screenshot. You'll see that a some of the keys are getting requests at a few thousand per second, but virtually all those requests are failing, causing a true throughput of close to zero.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        bengber bengber added a comment -

        From our side there is no immediate issue. In the future, we'll experiment with 2.2.0 and the settings you describe.

        Show
        bengber bengber added a comment - From our side there is no immediate issue. In the future, we'll experiment with 2.2.0 and the settings you describe.
        Hide
        dipti Dipti Borkar added a comment -

        Thanks Michael. Given that we have fixed this issue in 1.2 of the Java client, we can resolve this bug and Ben can re-open if we aren't comparable with other databases.

        In addition to this both 2.1 includes the multiple readers and writers that improved persistence to disk speeds. 3.0 includes the global i/o manager which we have seen significantly further improves persisting the disk.
        These enhancements will help as well.

        Show
        dipti Dipti Borkar added a comment - Thanks Michael. Given that we have fixed this issue in 1.2 of the Java client, we can resolve this bug and Ben can re-open if we aren't comparable with other databases. In addition to this both 2.1 includes the multiple readers and writers that improved persistence to disk speeds. 3.0 includes the global i/o manager which we have seen significantly further improves persisting the disk. These enhancements will help as well.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Matt,

        can you take a look at this issue?

        Show
        maria Maria McDuff (Inactive) added a comment - Matt, can you take a look at this issue?
        Hide
        anil Anil Kumar added a comment -

        Fixed this issue in 1.2 of the Java client, resolve this bug please re-open if still not acceptable.

        Show
        anil Anil Kumar added a comment - Fixed this issue in 1.2 of the Java client, resolve this bug please re-open if still not acceptable.
        Hide
        ingenthr Matt Ingenthron added a comment -

        Thanks for the update Anil, I didn't see it had been assigned to me.

        Show
        ingenthr Matt Ingenthron added a comment - Thanks for the update Anil, I didn't see it had been assigned to me.

          People

          • Assignee:
            ingenthr Matt Ingenthron
            Reporter:
            bengber bengber
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes