Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58536

DCP replication is hung with some nodes consistently TMP_OOM-ing while below HWM

    XMLWordPrintable

Details

    Description

      1. create a 3 KV node cluster
      2. create a 1024MB magma bucket with FULL eviction policy set.
      3. create 2 collections and start insert mutations.
      4. after some 3M items the mutations are stopped. Increase the RAM quota to 2048 and mutations started again.
      5. Reduce the ram quota back to 1024MB and items started evicting but mutations aren't going through due to etmpfail. DCP replication is also hung.

      Errors from server:

      {"error":"retry_needed","reason":"etmpfail returned from memcached"}

      Document mutations are failing through UI:

      Attachments

        1. image-2023-09-06-13-51-45-609.png
          55 kB
          Ritesh Agarwal
        2. image-2023-09-06-13-56-52-409.png
          450 kB
          Ritesh Agarwal
        3. image-2023-09-06-15-45-05-332.png
          193 kB
          Ritesh Agarwal
        4. image-2023-09-06-15-49-54-609.png
          322 kB
          Ritesh Agarwal
        5. image-2023-09-06-16-20-40-716.png
          390 kB
          Ritesh Agarwal
        6. image-2023-09-06-16-26-12-329.png
          792 kB
          Ritesh Agarwal
        7. image-2023-09-07-17-21-56-143.png
          64 kB
          Vesko Karaganev
        8. image-2023-09-07-17-22-13-665.png
          78 kB
          Vesko Karaganev
        9. image-2023-09-07-17-22-32-733.png
          84 kB
          Vesko Karaganev
        10. image-2023-09-11-09-50-38-573.png
          45 kB
          Vesko Karaganev
        11. image-2023-09-11-09-51-10-754.png
          55 kB
          Vesko Karaganev
        12. Screenshot_20230907_165833.png
          84 kB
          Vesko Karaganev
        13. Screenshot_20230908_155311.png
          108 kB
          Vesko Karaganev
        14. screenshot-1.png
          386 kB
          Ritesh Agarwal
        15. screenshot-2.png
          295 kB
          Ritesh Agarwal
        16. screenshot-3.png
          331 kB
          Ritesh Agarwal

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-58463 Replication stuck caused by Checkpoint OOM

          • Test Blocker - QE-related tests can't proceed
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty