Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29879

memcached is OOM-killed during data ingestion due to high heap fragmentation

    XMLWordPrintable

Details

    Description

      Build 5.5.0-2814.

      Setup:

      • 2 KV nodes + 4 N1QL & GSI nodes
      • 32GB RAM, 20GB memory quota
      • 1 bucket, 1 replica, full ejection

      Steps:

      /opt/couchbase/bin/cbbackupmgr restore --archive /backups --repo bigfun20M --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes

      That step restores about 320M relatively small docs.

      It looks like memcached RSS exceeds the memory quota more than I would normally expect.

      Attachments

        1. 172.23.96.5-defragthreshold=1-defraginterval=0.png
          172.23.96.5-defragthreshold=1-defraginterval=0.png
          152 kB
        2. 172.23.96.7-defragthreshold=1-defraginterval=0.png
          172.23.96.7-defragthreshold=1-defraginterval=0.png
          151 kB
        3. age10-varyinterval0.5-0.25-0.1-cpu.png
          age10-varyinterval0.5-0.25-0.1-cpu.png
          61 kB
        4. image.png
          image.png
          151 kB
        5. mem_usage.png
          mem_usage.png
          243 kB
        6. mem-5_replica-checkpoint-detail.png
          mem-5_replica-checkpoint-detail.png
          237 kB
        7. mem-5.png
          mem-5.png
          172 kB
        8. mem-7.png
          mem-7.png
          150 kB
        9. Screen Shot 2018-06-01 at 12.50.59.png
          Screen Shot 2018-06-01 at 12.50.59.png
          57 kB
        10. Screen Shot 2018-06-01 at 12.51.09.png
          Screen Shot 2018-06-01 at 12.51.09.png
          66 kB
        11. Screen Shot 2018-06-01 at 14.52.42.png
          Screen Shot 2018-06-01 at 14.52.42.png
          79 kB

        Issue Links

          is triggering

          Bug - A problem which impairs or prevents the functions of the product. MB-30017 High vb_replica_checkpoint_memory when handling slow streams on the Producer during data ingestion

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-29928 Defragmenter: Only run when fragmentation is high

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-15009 Defragment document key+meta (i.e. StoredValue) in addition to Blobs

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty