Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58536

DCP replication is hung with some nodes consistently TMP_OOM-ing while below HWM

    XMLWordPrintable

Details

    Description

      1. create a 3 KV node cluster
      2. create a 1024MB magma bucket with FULL eviction policy set.
      3. create 2 collections and start insert mutations.
      4. after some 3M items the mutations are stopped. Increase the RAM quota to 2048 and mutations started again.
      5. Reduce the ram quota back to 1024MB and items started evicting but mutations aren't going through due to etmpfail. DCP replication is also hung.

      Errors from server:

      {"error":"retry_needed","reason":"etmpfail returned from memcached"}

      Document mutations are failing through UI:

      Attachments

        1. image-2023-09-06-13-51-45-609.png
          image-2023-09-06-13-51-45-609.png
          55 kB
        2. image-2023-09-06-13-56-52-409.png
          image-2023-09-06-13-56-52-409.png
          450 kB
        3. image-2023-09-06-15-45-05-332.png
          image-2023-09-06-15-45-05-332.png
          193 kB
        4. image-2023-09-06-15-49-54-609.png
          image-2023-09-06-15-49-54-609.png
          322 kB
        5. image-2023-09-06-16-20-40-716.png
          image-2023-09-06-16-20-40-716.png
          390 kB
        6. image-2023-09-06-16-26-12-329.png
          image-2023-09-06-16-26-12-329.png
          792 kB
        7. image-2023-09-07-17-21-56-143.png
          image-2023-09-07-17-21-56-143.png
          64 kB
        8. image-2023-09-07-17-22-13-665.png
          image-2023-09-07-17-22-13-665.png
          78 kB
        9. image-2023-09-07-17-22-32-733.png
          image-2023-09-07-17-22-32-733.png
          84 kB
        10. image-2023-09-11-09-50-38-573.png
          image-2023-09-11-09-50-38-573.png
          45 kB
        11. image-2023-09-11-09-51-10-754.png
          image-2023-09-11-09-51-10-754.png
          55 kB
        12. Screenshot_20230907_165833.png
          Screenshot_20230907_165833.png
          84 kB
        13. Screenshot_20230908_155311.png
          Screenshot_20230908_155311.png
          108 kB
        14. screenshot-1.png
          screenshot-1.png
          386 kB
        15. screenshot-2.png
          screenshot-2.png
          295 kB
        16. screenshot-3.png
          screenshot-3.png
          331 kB

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-58463 Replication stuck caused by Checkpoint OOM

          • Test Blocker - QE-related tests can't proceed
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty