Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7463

Not possible to insert data much larger than memory

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Four node cluster. Each node: 32 GB of RAM, 8 Xeon cores, RAID0 of 4 SSD as a disk storage.
      RAM Quota 25720 MB, one bucket with 1 replica.

      Description

      Using YCSB we tried to load into this cluster the 500M of 120 byte records.

      The inserts came on a high rate (more than 200k per sec) while there is enough RAM. Then the insert rate dramatically decreased, the client got Temporary failure errors. This lasted for some hours and then, after inserting about 200M of objects the ejections, creation of new objects and disk writes stopped.

      We tried to limit the insert rate with 10k per second. This value is lower than the drain rate observed on the initial phase of the experiment (when there is enough RAM). But the issue is repeated after inserting of 400M of objects. The drain rate came to zero, ejections and creations stopped.

      1. drain_rate_with_10k_inserts.png
        556 kB
        Denis Nelubin
      2. drain_rate_with_unlimited_inserts.png
        390 kB
        Denis Nelubin
      3. failure.png
        549 kB
        Denis Nelubin
      4. mamory_usage.png
        235 kB
        Denis Nelubin
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        dnelubin Denis Nelubin added a comment -

        We found that in most cases Couchbase doesn't want to eject user data from memory to free it for metadata of new (active or replica) items.
        I.e. there are cases when it's not possible to insert one more record while 60% of memory is used by the user data.

        Show
        dnelubin Denis Nelubin added a comment - We found that in most cases Couchbase doesn't want to eject user data from memory to free it for metadata of new (active or replica) items. I.e. there are cases when it's not possible to insert one more record while 60% of memory is used by the user data.
        Hide
        mikew Mike Wiederhold added a comment -

        Dennis,

        I looked at this issue further and found that your meta-data overhead was high, but not high enough to cause the behavior you observed. What I really need now to address this issue further would be to reproduce this scenario so I can look at a live cluster. In order to do this I have filed an issue with our QE team to add this use case as well as some others like it so that we can catch bulk loading related issues. I have filed a ticket, CBQE-1078, in order to track this issue. We will also add a feature soon that reduces meta data overhead (CBD-569) which will further help solve this issue. Thanks for being patient and following up with us since this has helped greatly in understanding this issue. For now I am going to close this as a duplicate of the two issues I mentioned in this comment. Please feel free to follow up on this ticket of you have any other questions or you thing this issue should still be left open.

        Show
        mikew Mike Wiederhold added a comment - Dennis, I looked at this issue further and found that your meta-data overhead was high, but not high enough to cause the behavior you observed. What I really need now to address this issue further would be to reproduce this scenario so I can look at a live cluster. In order to do this I have filed an issue with our QE team to add this use case as well as some others like it so that we can catch bulk loading related issues. I have filed a ticket, CBQE-1078, in order to track this issue. We will also add a feature soon that reduces meta data overhead ( CBD-569 ) which will further help solve this issue. Thanks for being patient and following up with us since this has helped greatly in understanding this issue. For now I am going to close this as a duplicate of the two issues I mentioned in this comment. Please feel free to follow up on this ticket of you have any other questions or you thing this issue should still be left open.
        Hide
        mikew Mike Wiederhold added a comment -

        Actually, I'm going to run some tests myself before officially closing this as a duplicate.

        Show
        mikew Mike Wiederhold added a comment - Actually, I'm going to run some tests myself before officially closing this as a duplicate.
        Hide
        maria Maria McDuff (Inactive) added a comment - - edited

        dupe of MB-7518

        Show
        maria Maria McDuff (Inactive) added a comment - - edited dupe of MB-7518
        Hide
        maria Maria McDuff (Inactive) added a comment -

        this is because of bottleneck while writing to disk, mwr will improve this (in 2.0.2 release).

        Show
        maria Maria McDuff (Inactive) added a comment - this is because of bottleneck while writing to disk, mwr will improve this (in 2.0.2 release).

          People

          • Assignee:
            abhinav Abhinav Dangeti
            Reporter:
            dnelubin Denis Nelubin
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes