Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58111

Memcached crashes when mem HWM hit for ephemeral buckets in 7.6.0

    XMLWordPrintable

Details

    • Untriaged
    • Ubuntu 64-bit
    • 0
    • Unknown

    Description

      We are seeing issues in our KV "cache" perf test with 7.6.0. The test measures avg. ejections per second from a single ephemeral bucket, and with 7.6.0 we are seeing a reported 0 ejections per second:

      Offending test runs:

      1. http://perf.jenkins.couchbase.com/job/ares/36623/
      2. http://perf.jenkins.couchbase.com/job/ares/36647/
      3. http://perf.jenkins.couchbase.com/job/ares/36649/console (logs from this run: 172.23.133.13.zip - Update: there are no less than 5 crash dumps in here - presumably one for each time we lose all the items)

       

      When looking at the cbmonitor comparison report between 7.5.0-4421 and 7.6.0-1295, we see that in 7.6 the curr_items is reduced to 0 several times during the test:

      Examining the cluster during a test shows us that when we reach high mem watermark, in 7.6 we just seem to get rid of all active data and take no account of the low mem watermark:

       

      The second issue (possibly a test issue, but not sure) is that the stats we are using to measure ejection rate are no longer working it seems. In our test we query the following stats directly from memcached (using a mcbp client):

      • vb_active_auto_delete_count
      • vb_pending_auto_delete_count
      • vb_replica_auto_delete_count

      It seems these are all just 0 the whole time.

       

      Update: Couchbase Server crashes every time we reach high mem watermark it seems, so its a bit more of a serious issue...

      Attachments

        1. image-2023-08-02-20-11-41-034.png
          39 kB
          Daniel Nagy
        2. image-2023-08-02-20-10-12-027.png
          70 kB
          Daniel Nagy
        3. image-2023-08-02-20-04-21-558.png
          35 kB
          Daniel Nagy

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Daniel.nagy Daniel Nagy
              Daniel.nagy Daniel Nagy
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty