Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27933

Couchbase doesn't accept writes from Sync Gateway due to bucket OOM

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • couchbase-bucket
    • Build: 5.5.0-1812

      Cluster: hebe OS: CentOS 7
      CPU: E5-2680 v3 (48 vCPU)
      Memory: 15GB data, 45GB index
      Disk: Samsung PM863a
    • Untriaged
    • Unknown

    Description

      It behaves like xattrs aren't being evicted although the bucket is in "Full eviction" mode.

      Setup:
      4 node cluster, 15GB RAM per node for data
      Full eviction mode, 1 replica

       

      I'm loading documents, every document is about 30KB and has about 43KB long xattr object.
      After 900K documents have been loaded the server stops accepting writes and throws 404 with "Server out of memory" error

      Cluster stats at that moment:

      High water mark = 49.8 GB
      Data size = 32.8GB
      Resident Ratio = 0.03%
      User Data in RAM (Active, Replica) = 150MB
      User Data in RAM (Pending) = 0
      User Data in RAM (Total) = 44.8GB
      Ejections per second =  0

       

      Attachments

        1. Bucket Config.png
          Bucket Config.png
          76 kB
        2. Data Size.png
          Data Size.png
          7 kB
        3. Main.go
          0.9 kB
        4. quick_drop1.png
          quick_drop1.png
          32 kB
        5. quick_drop2.png
          quick_drop2.png
          97 kB
        6. vBucket Details.png
          vBucket Details.png
          53 kB
        7. xattr_oom_cbcollect.node190.zip
          6.85 MB
        8. xattr_oom_cbcollect.node191.zip
          6.62 MB
        9. xattr_oom_cbcollect.node192.zip
          6.70 MB
        10. xattr_oom_cbcollect.node193.zip
          6.24 MB
        11. xattr_oom_cbcollect.zip
          11.30 MB

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-29227 Couchbase Server can get into a livelock state due to high checkpoint memory usage

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-29189 Items get stuck in replication queue during initial data load

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              oleksandr.gyryk Alex Gyryk (Inactive)
              oleksandr.gyryk Alex Gyryk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty