Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58111

Memcached crashes when mem HWM hit for ephemeral buckets in 7.6.0

    XMLWordPrintable

Details

    • Untriaged
    • Ubuntu 64-bit
    • 0
    • Unknown

    Description

      We are seeing issues in our KV "cache" perf test with 7.6.0. The test measures avg. ejections per second from a single ephemeral bucket, and with 7.6.0 we are seeing a reported 0 ejections per second:

      Offending test runs:

      1. http://perf.jenkins.couchbase.com/job/ares/36623/
      2. http://perf.jenkins.couchbase.com/job/ares/36647/
      3. http://perf.jenkins.couchbase.com/job/ares/36649/console (logs from this run: 172.23.133.13.zip - Update: there are no less than 5 crash dumps in here - presumably one for each time we lose all the items)

       

      When looking at the cbmonitor comparison report between 7.5.0-4421 and 7.6.0-1295, we see that in 7.6 the curr_items is reduced to 0 several times during the test:

      Examining the cluster during a test shows us that when we reach high mem watermark, in 7.6 we just seem to get rid of all active data and take no account of the low mem watermark:

       

      The second issue (possibly a test issue, but not sure) is that the stats we are using to measure ejection rate are no longer working it seems. In our test we query the following stats directly from memcached (using a mcbp client):

      • vb_active_auto_delete_count
      • vb_pending_auto_delete_count
      • vb_replica_auto_delete_count

      It seems these are all just 0 the whole time.

       

      Update: Couchbase Server crashes every time we reach high mem watermark it seems, so its a bit more of a serious issue...

      Attachments

        1. 172.23.133.13.zip
          20.75 MB
        2. 760_1007_172.23.133.13.zip
          22.72 MB
        3. 760_1079_172.23.133.13.zip
          23.10 MB
        4. 760_1189_172.23.133.13.zip
          23.19 MB
        5. 760_1242_172.23.133.13.zip
          23.36 MB
        6. 760_1266_172.23.133.13.zip
          22.48 MB
        7. 760_1271_172.23.133.13.zip
          22.94 MB
        8. 760_1274_172.23.133.13.zip
          19.07 MB
        9. 760_1280_172.23.133.13.zip
          19.03 MB
        10. image-2023-08-02-20-04-21-558.png
          image-2023-08-02-20-04-21-558.png
          35 kB
        11. image-2023-08-02-20-10-12-027.png
          image-2023-08-02-20-10-12-027.png
          70 kB
        12. image-2023-08-02-20-11-41-034.png
          image-2023-08-02-20-11-41-034.png
          39 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Daniel.nagy Daniel Nagy
              Daniel.nagy Daniel Nagy
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty