Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4292

ep-engine does not flush expired replica items if expiry time is 2-3 seconds (memory leak)

    Details

      Description

      to reproduce :

      add 2k items with expiry = 0
      then add 10k items each 50k where expiry is set to 2 seconds
      wait until expiry pager runs( you can modify the expiry pager time to 120 seconds instead of the default which is 60 minutes)

      then look at the active and replica items:

      in my case : active items : 2000 , replica items on node 1: 6700 and node2: 7800 items

      i used mbstats hash to get the number of items in acrive and replica vbucket :

      vb_38 is active on node 10.1.5.38 and replica in 10.1.5.54

      vb_38:counted is 1 for the active node versus 16 on the replica

      root@ubuntu1004-64:~# /opt/membase/bin/mbstats 10.1.5.54:11210 hash detail | grep vb_38:
      vb_38:counted: 16
      vb_38:histo_0,1: 3063
      vb_38:histo_1,2: 16
      vb_38:locks: 5
      vb_38:max_depth: 1
      vb_38:mem_size: 2288
      vb_38:mem_size_counted: 2288
      vb_38:min_depth: 0
      vb_38:reported: 16
      vb_38:resized: 0
      vb_38:size: 3079
      vb_38:state: replica
      root@ubuntu1004-64:~# /opt/membase/bin/mbstats 10.1.5.38:11210 hash detail | grep vb_38:
      vb_38:counted: 1
      vb_38:histo_0,1: 3078
      vb_38:histo_1,2: 1
      vb_38:locks: 5
      vb_38:max_depth: 1
      vb_38:mem_size: 143
      vb_38:mem_size_counted: 143
      vb_38:min_depth: 0
      vb_38:reported: 1
      vb_38:resized: 0
      vb_38:size: 3079
      vb_38:state: active
      root@ubuntu1004-64:~#

      because of this the memory usage also remains higher than it should be :

      32613 membase 20 0 1248m 1.1g 3088 S 2 45.8 2:43.53 memcached

      root@ubuntu1004-64:~# /opt/membase/bin/mbstats 10.1.5.54:11210 all | grep mem_
      ep_mem_high_wat: 943718400
      ep_mem_low_wat: 754974720
      mem_used: 27999326

      mem_used is 279MB
      if i restart the node mem_used drops to 60 meg and memcached resident memory also becomes 70 MB ( down from 1.1 GB )

      after restart active and replica items count will also drop to 1000

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            • Assignee:
              chiyoung Chiyoung Seo (Inactive)
              Reporter:
              farshid Farshid Ghods (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes