Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8084

Couchbase stops ejecting keys. Returns nothing but Temp OOM errors until restarted.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0.1
    • Fix Version/s: None
    • Component/s: None
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Ubuntu 11.10

      Description

      I have a single node Couchbase 2.0 server, 16gb of RAM allocated to the cluster, with a bucket that expects ~650,000 new key/values a day. Each key/value around 100-200KB.
      It seems though that after around 16GB of new writes, the server stops ejecting keys. The "memory used" goes beyond the low watermark (12.8GB) and beyond the high watermark as well (14.5GB) and never backs down.

      A disk write issue seems likely, but the disk write queue is empty when the Temp OOM errors start showing up.

      I've tried tweaking the low and high water mark values. I've also set the min_data_age to 0.
      Still with all this, it after about 16GB of new writes, restarting the server is the only thing that will make it work again.

      Here is a screen-cap of the Bucket Analytics where the problem is visible.
      https://dl.dropbox.com/u/171745/Couchbase-Console-%282.0.1%29.jpg

      I'm attached the cbcollect_info for the server.

      Issue created because of the following request:
      http://www.couchbase.com/forums/thread/couchbase-stops-ejecting-keys-returns-nothing-temp-oom-errors-until-restarted#comment-1009432

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        mikew Mike Wiederhold added a comment -

        Great, let me know if you run into any other problems.

        Show
        mikew Mike Wiederhold added a comment - Great, let me know if you run into any other problems.
        Hide
        mikew Mike Wiederhold added a comment -

        Brent,

        Do you have any update on this? If there's a bug here I want to try to get it in the next release so please let me know.

        Thanks,

        • Mike
        Show
        mikew Mike Wiederhold added a comment - Brent, Do you have any update on this? If there's a bug here I want to try to get it in the next release so please let me know. Thanks, Mike
        Hide
        brentley_11 Brent Housen added a comment -

        Sorry, I haven't had time to assess the situation at all. Something more dire has come up. Haha. Our host recently contacted mentioning a memory leak that fixed. On initial glance it actually looks like that could of been causing our problems. I'm not sure if I'll have time to look into this further at all this week, so maybe it's best to just close this ticket and I'll open again if I can duplicate the problem.

        Show
        brentley_11 Brent Housen added a comment - Sorry, I haven't had time to assess the situation at all. Something more dire has come up. Haha. Our host recently contacted mentioning a memory leak that fixed. On initial glance it actually looks like that could of been causing our problems. I'm not sure if I'll have time to look into this further at all this week, so maybe it's best to just close this ticket and I'll open again if I can duplicate the problem.
        Hide
        mikew Mike Wiederhold added a comment -

        Thanks for following up. I'll close it now as "cannot reproduce". If you run into the issue again please reopen it and I will investigate the problem. If there is a bug we can get a fix in for whatever the next release is.

        Show
        mikew Mike Wiederhold added a comment - Thanks for following up. I'll close it now as "cannot reproduce". If you run into the issue again please reopen it and I will investigate the problem. If there is a bug we can get a fix in for whatever the next release is.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        will re-open if cust repros the issue.

        Show
        maria Maria McDuff (Inactive) added a comment - will re-open if cust repros the issue.

          People

          • Assignee:
            mikew Mike Wiederhold
            Reporter:
            brentley_11 Brent Housen
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes